The nature of the building blocks of information in visual working memory (VWM) is a fundamental issue that has not been well resolved. Most researchers take objects as the building blocks, although this perspective has received criticism. The objects could be physically separated ones (strict object hypothesis) or hierarchical objects created from separated individuals (broad object hypothesis). Meanwhile, a newly proposed Boolean map theory for visual attention suggests that Boolean maps may be the building blocks of VWM (Boolean map hypothesis); this perspective could explain many critical findings of VWM. However, no previous study has examined these hypotheses. We explored this issue by focusing on a critical point on which they make distinct predictions. We asked participants to remember two distinct objects (2-object), three distinct objects (3-object), or three objects with repeated information (mixed-3-object, e.g., one red bar and two green bars, green bars could be represented as one hierarchical object) and adopted contralateral delay activity (CDA) to tap into the maintenance phase of VWM. The mixed-3-object condition could generate two Boolean maps, three objects, or three objects most of the time (hierarchical objects are created in certain trials, retaining two objects). Simple orientations (Experiment 1) and colors (Experiments 2 and 3) were used as stimuli. Although the CDA of the mixed-3-object condition was slightly lower than that of the 3-object condition, no significant difference was revealed between them. Both conditions displayed significantly higher CDAs than the 2-object condition. These findings support the broad object hypothesis. We further suggest that Boolean maps might be the unit for retrieval/comparison in VWM.