The museum was designed by architects Guy Lagneau, Michel Weill, Jean Dimitrijevic and Raymond Audigier, who worked closely with engineers Bernard Laffaille and René Sarger for the concrete and steel structural work, Jean Prouvé for the aluminum elements, and André Salomon for the lighting design, which comprised both natural and artificial sources. The architects also commissioned Henri-Georges Adam to create a monumental sculpture for the building's facade. The innovative architectural design was based on modularity and transparency—principles that the pioneering team of architects and engineers translated into a building that reflected the desire for something distinctively modern expressed by Georges Salles, then Director of Musées de France, and Reynold Arnould, Curator of the city's museums.

The museum is located at the tip of a residential neighbourhood typical of Le Havre's reconstruction architecture. Its design breaks with both reconstruction architect Auguste Perret's signature style and—perhaps most of all—with the aesthetics traditionally associated with this type of institution.
The architects' model for the future building was unveiled in December 1953 at Paris' Musée national d'art moderne during an exhibition of the Le Havre museum's collection of works from Corot to the present. However, ground was not broken on the new building until 1958. In the meantime, plans for the museum had taken shape and its purpose and design determined. When André Malraux, then France's Minister of Cultural Affairs, inaugurated the museum on June 24, 1961, it had evolved into a full-fledged cultural centre.

The building's design clearly expressed the museum's avant-garde vocation, as did its approach to exhibition design. Works were shown in new ways that radically transformed how the collections were seen. The modular exhibition halls were designed to meet the needs of a constantly evolving cultural programme that included exhibitions, of course, but also concerts, lectures and performances.

The Musée d'art moderne André Malraux was designed as a smooth, transparent assembly of glass and steel perched on a concrete base, anchoring it firmly on the sea-facing site.
The reinforced-concrete platform that makes up the lower level is devoted to the building's foundations and storage for works not on exhibition. The upper level is framed in steel. The building is glassed-in on five sides, letting in abundant natural light. The southern and western facades, exposed to the wind, are built from two expanses of glass and aluminum panels designed by Jean Prouvé. Prouvé also designed the impressive 7-metre-by-6-metre door that serves as a service entrance on the western side of the building and the arbour-like rooftop screen. The aluminum blades of the monumental louvered screen—a prime example of the building's advanced engineering—evoke the wings of an aircraft, filtering the sunlight and creating a soft, natural glow throughout the building's interior.

And light is the museum's most striking element. Natural light, traditionally a key component of most museums built in the 20th century, is of course present, but it is complemented by laterally-directed artificial light emanating from the four cardinal directions—north, south, east, and west—rather than from the north alone, which is typical of more traditional architectural designs. On the eastern facade, semi-opaque glass softens the morning sun, while the western facade is equipped with three light-filtering structures: an etched-glass wall with strong horizontal lines, vertical louvered shutters, and, when the sun's rays are at their most intense, blinds. Square translucent ceiling panels filter the light reflected by the rooftop screen.

The museum's interior is free from load-bearing walls, making the space completely modular. Built on just a few pillars, the structure encases a huge volume that can be set up in a variety of ways for a multitude of activities and events. The open floor plan and 550 square metres of glass surfaces give the entire space a transparent look and feel. The changing light of the Seine River estuary—the inspiration for so many of the works within the museum—is omnipresent.
The on-going conversation between the museum's works and the maritime landscape that inspired them offers visitors what Raoul Dufy demanded for himself: "The artist needs what he sees to be bathed in a certain quality of light, a shimmering, almost-breathing light."

The 1995–1999 museum renovation project was a joint effort between the architects, Laurent and Emmanuelle Beaudouin, Jean-Pierre Crousse, and Sandra Barclay, a team of engineers, and the museum's curator, Françoise Cohen.

The project aimed to reorganize the space around a central atrium, give the collections a fresh new backdrop, offer visitors a pleasant, coherent pathway through the museum and maintain the visual link with the marine landscape—all without fundamentally altering the building's architectural essence.

The picture rails are kept relatively low, allowing the eye to travel unhindered from one work to another and from one area of the museum to another. The large central atrium is used to show works from the permanent collections and for temporary exhibitions, while the smaller areas on the mezzanine level are used to display the museum's older collections. All of this creates a visual and spatial rhythm, one that is reflected in the ceiling materials—clear glass in the atrium and around the periphery of the building and opaque on the mezzanine level. The two levels used for exhibitions are linked by an elevated walkway that gradually reveals the museum's relationship to the landscape. Substantial engineering work went into renovating the exterior facades. To better diffuse the natural sunlight on the western side of the building, etched glass louvers were installed between the two facade walls.

In 2006, after the museum received the works in the Senn-Foulds donation, architects Laurent Beaudouin and Sandra Barclay reorganized the eastern side of the mezzanine level to house the new collection. The amount of wall space was increased to accommodate a larger number of paintings, but the space was left relatively open to let the eye travel unhindered and to ensure the circulation of light. A new skylight was added, and the visual relationship between the building and landscape underscored. The drawing gallery was also redesigned.

Henri-Georges Adam's monumental sculpture The Signal was commissioned by the French government in 1956. It is an integral part of the museum and has left a strong mark on the museum's identity.
The 22-metre-long, 7-metre-high, 220-tonne sculpture reveals a fragment of the landscape, creating a concrete frame and underscoring the building's figurehead-like location at the entrance to the port of Le Havre.

The sculpture's French title, Le Signal [The Signal], which describes its purpose, does not appear to have been given by the artist initially. The title probably stood out as the project evolved, and the work is catalogued under the name Le Signal by the national contemporary art collection and the museum. The people of Le Havre, however, spontaneously refer to it under several names, including L'oeil [The Eye], La Navette [The Shuttle], and La Boussole [The Compass].

Over five decades of unprotected exposure to dominant winds and harsh weather—a major cause of erosion—ultimately took their toll on the monumental sculpture. The restoration that was carried out for the museum's 50th anniversary returned the work to its original state, reviving the ties between the museum, the port and the sea beyond.