Map with TTL option in Go

Question

I need to build a data-structure like this:

map[string]SomeType 

But it must store values for about 10 minutes and then clear it from memory. Second condition is records amount - it must be huge. This data-structure must add at least 2-5K records per second.

So, what is the most correct way in Go to make it?

I'm trying to make goroutine with timeout for each new elemnt. And one(or more) garbage-collector goroutine with channel to receive timeouts and clear elements. But I'm not sure it's the most clear way. Is it Ok to have millions of waiting goroutines with timeouts?

Thanks.

Answer 1

You will have to create a struct to hold your map and provide custom get/put/delete funcs to access it.

Note that 2-5k accesses per second is not really that much at all, so you don't have to worry about that.

Here's a simple implementation:

type item struct {
    value      string
    lastAccess int64
}

type TTLMap struct {
    m map[string]*item
    l sync.Mutex
}

func New(ln int, maxTTL int) (m *TTLMap) {
    m = &TTLMap{m: make(map[string]*item, ln)}
    go func() {
        for now := range time.Tick(time.Second) {
            m.l.Lock()
            for k, v := range m.m {
                if now.Unix() - v.lastAccess > int64(maxTTL) {
                    delete(m.m, k)
                }
            }
            m.l.Unlock()
        }
    }()
    return
}

func (m *TTLMap) Len() int {
    return len(m.m)
}

func (m *TTLMap) Put(k, v string) {
    m.l.Lock()
    it, ok := m.m[k]
    if !ok {
        it = &item{value: v}
        m.m[k] = it
    }
    it.lastAccess = time.Now().Unix()
    m.l.Unlock()
}

func (m *TTLMap) Get(k string) (v string) {
    m.l.Lock()
    if it, ok := m.m[k]; ok {
        v = it.value
        it.lastAccess = time.Now().Unix()
    }
    m.l.Unlock()
    return

}

playground

note(2020-09-23): for some reason the time resolution on the current version of the playground is way off, this works fine, however to try on the playground you have to change the sleep to 3-5 seconds.