A mock up of the earlier Church using model housed in Pavey Rooms at the Elms in Charmouth.

In about 1503 Charmouth Church was rebuilt; and something is known of its structure, partly from descriptions given by Hutchins in his history of Dorset, and by Roberts, Lyme's historian; partly from the model shown in the church, and made from memory by W. Hoare of Charmouth in 1811; and partly from relics incorporated or housed in the present building. The model is in now in the Pavey Room Museum at the Parish Office in the 'Elms' for all to see, and may be taken as a fair representation of the building of ca. 1503. It had nave and chancel, a southern aisle, a northern door and porch, and a western tower. The Beer stonework forming the arch of the northern porch rests upon two carved heads which are evidently medieval; and, in that case, must have been saved from the old building. Inside, a minstrels' and singers ' gallery was placed in front of the tower, and in 1817, when a growing congregation demanded more room, another gallery was added along the northern side of the nave. The tower housed three bells, which were recast in 1836, and are still in use. The large Ellesdon Memorial stood in the church, and this and the mural monuments, and those on the floor, were transferred to the present building. In addition, some relics of great interest have been retrieved notably the stone figure of an Abbot, presumably an Abbot of Ford now standing in the south east comer and backed by a cross, which was found built in to the Rectory wall. It has been identified with a 'crucifix' mentioned by Roberts as crowning the church gable, and overgrown with ivy; but it is not a crucifix, and the ivy must have obscured Roberts' vision. Probably, as now, a plain cross surmounted the gable. The figure is made of Ham stone, and there is evidence that some of that material had been used in the church.
Nothing is known of the history of the called 'font,' now placed inside the northern door. It was left in the churchyard unti11895, and then brought inside the church. This is generally assumed to have been the functional font of the former church. It is remarkable for the small bowl and disproportionately stout and composite stem .The bowl, plinth and attached parts of the stem are made of a white stone - apparently Beer stone - while between them is inserted a stout cylinder of yellow Ham stone. Its disproportion, combined with its hybrid make-up, suggest improvisation, unlikely in a church in which worship had been conducted for centuries.
The chancel was divided from the nave by an oaken screen, of which a portion was found propping a clothesline in a cottage garden. It is now incorporated in the chimney-piece of the old Manor house, opposite the church. No remnant has been found of the nine grotesque carvings in oak, mentioned by Roberts, inside the seats by the screen, or of the pictures of the twelve apostles, six on each side of an Ecce Homo, painted in 1817 by the churchwarden, Lieut. Gabriel Bray, R.N., on the front of the new northern gallery. Lost, too, is the Wadham coat of arms, which is recorded as hanging in the church in 1600. William Wadham lived at Catherston and in 1590, acquired Newlands, now known a Stonebarrow Manor, which was then part of Charmouth Manor. His kinsman, Nicholas, founded Wadham College, Oxford. The high-backed oaken pews, now replaced were transferred to the new church, as well as the old pulpit of carved oak, which was placed in the middle of the nave.
In 1835, it was decided to enlarge the church by building an aisle on the northern side. The roof, however, was found to be in a most dilapidated and dangerous state. The architect, Charles Wallis, of Dorchester, reported that 'in the course of my practice and long experience, I have never witnessed so dilapidated or unsafe a building.' On the 29th January, 1835, the Vestry resolved that it was expedient to take down the present Church, and to build a new one.'
So perished the church dedicated to St. Matthew of ca. 1503, and our present church of St. Andrew began.