Login. Search. Navigation. Content

HERMES Project


Partnerarea


Forgot Password?

Logo EU 7th Framework Programme

Facilitation of Episodic Memory


It is 2 PM and Maria is at home having a little nap after lunch when the doorbell rings. She awakes at the second ring and goes to the door to see whos there. Through the door viewer she sees that it is Anna, her friend. She opens the door and leads Anna to the living room excusing herself for a few minutes. Her friend takes a seat on the sofa in Marias living room while Maria is freshening up. 

The HERMES audio and video sensors that are installed in Marias home recognize the doorbell as possible onset of activity and the HERMES system asks Maria whether she considers the event important enough to be permanently stored on the hard disk for processing and future retrieval. The system gives Maria 5 minutes to decide; during that time the outputs of the sensors are temporarily buffered by HERMES.

In this case she decides that she wants to store the event, she preses the corresponding button on her remote control before she is off for the bathroom. The HERMES system stores the buffered data and continues recording and saving the recording.

Anna sits there on the sofa waiting and reading one of the magazines that are lying on the couch table in front of her. Not much action is going on in the living room. The HERMES system is still recording following Marias categorization of the event as important, since the visual processing indicates at least one person present in the room.

Because Anna is a friend of Maria and has been at Marias place more than once the HERMES system recognizes her face. So later when Maria is searching for conversations with Anna the name is available in the system.

Maria is coming back from the bath room with some cookies and coffee. She takes a seat near Anna and they start to talk about this and that.

At about 5 pm Anna realizes that she has to return home. Maria escorts her to the door. The HERMES system notices that both are leaving the room and stops recording the event.

The HERMES system has stored the whole event from the first door bell ring to the exit of the two persons from the living room of Marias home. Also some contextual information, namely primary contextual cues were stored during the capture of the event. These are date, time and location of the event 

From this scenario, the on-line processing that needs to be performed is:

  • Detection of possible event onset: Audio recognition of doorbell ring, or some communication of the doorbell with the system.
  • Presence of people: Visual face detection in room and/or visual tracking of people.
  •  Detection of event end: Somehow the system must understand that the visitor has left, and stop recording, even if Maria remains in the room!

In the post processing that starts right after the end of the event secondary and if possible tertiary contextual cues are being generated by the HERMES system. These are:

  • Emotion in voice and/or face.
  • Identity of people present.
  • Transcription of the conversation

These annotations simplify the browsing and the retrieval of unique events from Marias life.

Retrieval

About two weeks later Maria is a little desperate. She knows that her friend Anna has been here but is not able to remember the topic of their conversation. So she consults the HERMES system and types in Annas name. She gets all conversations she had with Anna and chooses the right one upon the date. Maria can choose to either watch portions of the conversation or to read the conversation.