In the post since 2001, Mr Semel has been under pressure for some time amid disappointing trading results and the growing dominance of rival Google.
The firm suffered a 16% fall in profits in the first three months of 2007.
Mr Yang helped launch the business, which now has more than 500 million users worldwide, in 1995.
Recent struggles
Mr Semel will take on the role of non-executive chairman but will no longer be involved in the day-to-day running of the business.
Yahoo has struggled in recent years, losing market share to rivals such as Google and being criticised for poor technology and a lack of innovation.
It currently accounts for about 26% of all online searches in the US, well behind Google which has a market share above 49%.
Announcing his departure, Mr Semel said no-one at the firm was "satisfied" with the company's recent financial performance.
"This is a time for new executive leadership, with different skills and strengths, to step in and drive the company to realise its potential," he said.
"It is the right thing to do and the right time is now."
In a statement, Yahoo acknowledged that the past year had been a "difficult one" for the business.
But it stressed that it had made progress in key areas such as developing advertising search functions and building its presence in video and on mobile platforms.
Mr Yang, 38, created the original Yahoo search engine in 1995 alongside partner David Filo before helping to launch the business on the stock market a year later.
Since then, he has held a number of important roles overseeing corporate strategy, technological development and international expansion.
Microsoft link
Yahoo has repeatedly been linked with a deal with Microsoft to try and counter Google's increasing supremacy in the online search market.
The two were reported to have held informal merger talks last year and to have revived discussions over possible co-operation earlier this year.
The two firms have dismissed the reports as speculation but, on both occasions, such talk led to a sharp increase in Yahoo's share price.
Mr Semel joined Yahoo at a time when its fortunes were at a low ebb following the collapse of the dotcom boom and helped to lead a revival in its business.
But Google's formidable growth and concerns about Yahoo's ability to respond has led to growing doubts about the firm's direction.
Yahoo's shares rose nearly 3% after news of Mr Semel's departure was revealed.