#redis on irc.freenode.net 2011-01-12 (txt)

[00:11] brianmario (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) joined #redis.
[00:16] TomsB (~TomsB@46.109.55.4) left irc: Read error: Connection reset by peer
[00:27] mrflip_ (~mrflip@cpe-72-177-115-225.austin.res.rr.com) left irc: Quit: mrflip_
[00:31] brianmario (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) left irc: Quit: brianmario
[00:32] watsonian (~watsonian@c-76-103-10-132.hsd1.ca.comcast.net) left irc: Read error: Connection reset by peer
[00:35] nff (~nicolas@smtp-ox.owlient.eu) joined #redis.
[00:36] watsonian (~watsonian@c-76-103-10-132.hsd1.ca.comcast.net) joined #redis.
[00:38] mrflip_ (~mrflip@cpe-67-9-146-29.austin.res.rr.com) joined #redis.
[00:38] seivan (~seivan@109.228.150.197) left irc: Remote host closed the connection
[00:48] watsonian (~watsonian@c-76-103-10-132.hsd1.ca.comcast.net) left irc: Quit: Bye!
[00:57] horsey (~horsey@27.34.244.74) left irc: Quit: leaving
[00:59] xgmlovebee (74f612f8@gateway/web/freenode/ip.116.246.18.248) joined #redis.
[01:09] hikeonpast (~hikeonpas@71-95-209-242.static.mtpk.ca.charter.com) left irc: Ping timeout: 240 seconds
[01:10] nff (~nicolas@smtp-ox.owlient.eu) left irc: Ping timeout: 264 seconds
[01:12] TitusX (3ef59f0d@gateway/web/freenode/ip.62.245.159.13) joined #redis.
[01:13] hikeonpast (~hikeonpas@adsl-75-22-54-16.dsl.irvnca.sbcglobal.net) joined #redis.
[01:16] sechrist (~sechrist@c-76-102-99-133.hsd1.ca.comcast.net) joined #redis.
[01:17] TitusX (3ef59f0d@gateway/web/freenode/ip.62.245.159.13) left irc: Ping timeout: 265 seconds
[01:20] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[01:24] ttpva (~ttpva@a79-168-98-182.cpe.netcabo.pt) joined #redis.
[01:24] Wombert (~Wombert@zentrale.gutefrage.net) joined #redis.
[01:24] danlucraft (~Adium@79.173.154.114) joined #redis.
[01:25] nff (~nicolas@smtp-ox.owlient.eu) joined #redis.
[01:26] curtisc (~curtisc@99-8-186-153.lightspeed.snfcca.sbcglobal.net) joined #redis.
[01:26] deepthawtz (~deepthawt@c-24-4-229-52.hsd1.ca.comcast.net) left irc: Quit: Leaving...
[01:30] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[01:32] xgmlovebee (74f612f8@gateway/web/freenode/ip.116.246.18.248) left irc: Quit: Page closed
[01:41] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[01:42] drbobbeaty (~drbobbeat@c-67-184-75-162.hsd1.il.comcast.net) joined #redis.
[01:42] hikeonpast (~hikeonpas@adsl-75-22-54-16.dsl.irvnca.sbcglobal.net) left irc: Ping timeout: 240 seconds
[01:50] bzinger (~bzinger@beta.blubolt.com) joined #redis.
[01:51] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[01:56] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[01:58] kost_ (~kost@217.172.23.200) joined #redis.
[01:59] tmacedo (~tmacedo@90.163.57.2) joined #redis.
[02:02] curtisc (~curtisc@99-8-186-153.lightspeed.snfcca.sbcglobal.net) left irc: Quit: curtisc
[02:10] merlin83 (~merlin83@unaffiliated/merlin83) left irc: Ping timeout: 240 seconds
[02:11] kost_ (~kost@217.172.23.200) left irc: Read error: No route to host
[02:12] kost_ (~kost@217.172.23.200) joined #redis.
[02:12] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 265 seconds
[02:12] SpecialRuic (~anonymous@195.184.122.148) joined #redis.
[02:14] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[02:21] kost_ (~kost@217.172.23.200) left irc: Read error: No route to host
[02:21] kost_ (~kost@217.172.23.200) joined #redis.
[02:25] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[02:26] drbobbeaty (~drbobbeat@c-67-184-75-162.hsd1.il.comcast.net) left irc: Quit: drbobbeaty
[02:27] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) left irc: Quit: ank
[02:29] juangiordana_ (~quassel@190.31.86.179) joined #redis.
[02:31] juangiordana (~quassel@190.225.67.83) left irc: Ping timeout: 250 seconds
[02:53] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[02:59] lifeeth (~praneeth@unaffiliated/lifeeth) joined #redis.
[03:01] juangiordana_ -> juangiordana
[03:02] laurent2 (~laurent2@ks363276.kimsufi.com) joined #redis.
[03:02] richardb (richard@atreus.tartarus.org) joined #redis.
[03:16] atoi (~atoi@c-71-198-19-199.hsd1.ca.comcast.net) left irc: Quit: Leaving
[03:17] iFire` (~kittens@unaffiliated/ifire) left irc: Read error: Connection reset by peer
[03:20] iFire (~kittens@unaffiliated/ifire) joined #redis.
[03:25] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Quit: Leaving.
[03:30] kost_ (~kost@217.172.23.200) left irc: Remote host closed the connection
[03:36] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[03:38] Sunray (~noop@unaffiliated/sunray) joined #redis.
[03:40] matflores (~matias@190.50.97.110) joined #redis.
[03:43] drbobbeaty (~drbobbeat@38.98.137.29) joined #redis.
[03:50] elcuervo (~elcuervo@r186-48-221-215.dialup.adsl.anteldata.net.uy) joined #redis.
[03:54] danlucraft (~Adium@79.173.154.114) left irc: Read error: Connection reset by peer
[03:54] danlucraft1 (~Adium@79.173.154.114) joined #redis.
[03:55] lifeeth (~praneeth@unaffiliated/lifeeth) left irc: Quit: Up and at 'em, Atom Ant!
[04:00] merlin83 (merlin83@unaffiliated/merlin83) joined #redis.
[04:09] br1 (~BrunoM@r190-135-7-142.dialup.adsl.anteldata.net.uy) joined #redis.
[04:09] agentzh (~agentz@nginx/adept/agentzh) left irc: Quit: Leaving.
[04:14] tizoc (~user@unaffiliated/tizoc) joined #redis.
[04:14] KevBurnsJr (~KevBurnsJ@75.52.124.38) joined #redis.
[04:19] lifeeth (~praneeth@unaffiliated/lifeeth) joined #redis.
[04:28] merlin83 (merlin83@unaffiliated/merlin83) left irc: Ping timeout: 240 seconds
[04:28] lifeeth_ (~praneeth@202.3.77.208) joined #redis.
[04:28] lifeeth_ (~praneeth@202.3.77.208) left irc: Changing host
[04:28] lifeeth_ (~praneeth@unaffiliated/lifeeth) joined #redis.
[04:28] lifeeth (~praneeth@unaffiliated/lifeeth) left irc: Read error: Connection reset by peer
[04:33] merlin83 (merlin83@unaffiliated/merlin83) joined #redis.
[04:34] thinkingpotato (~thinkingp@abbn74.neoplus.adsl.tpnet.pl) joined #redis.
[04:40] DubLo7 (~Adium@24-247-75-139.dhcp.trcy.mi.charter.com) left irc: Quit: Leaving.
[04:44] horsey (~horsey@27.34.244.74) joined #redis.
[04:53] <horsey> I am using redis-py to store a python dict in Redis. However, I am not able to retrieve what I set as a python dict. However, when I fetch it again, it is returned to me as a string. Is there a way I can get a dict?
[05:00] lifeeth_ -> lifeeth
[05:00] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) joined #redis.
[05:08] d0k (~d0k@aef.wh.uni-dortmund.de) joined #redis.
[05:08] d0k_ (~d0k@aef.wh.uni-dortmund.de) joined #redis.
[05:08] DubLo7 (~Adium@71-83-3-210.static.aldl.mi.charter.com) joined #redis.
[05:10] d0k (~d0k@aef.wh.uni-dortmund.de) left irc: Client Quit
[05:10] d0k_ (~d0k@aef.wh.uni-dortmund.de) left irc: Client Quit
[05:10] d0k (~d0k@aef.wh.uni-dortmund.de) joined #redis.
[05:13] <nff> horsey, redis can only store strings. you have to check if your client library supports automatic serialisation or use pickle by hand if it doesn't
[05:14] <bz> horsey: the redis-py driver supports __{set,get}attr__. so the redis.Redis() acts like a dict.
[05:16] <bz> horsey: um, i misread your question. if you're going to store an instance of dict, couldn't you use the loads/dumps functions from the pickle module?
[05:18] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[05:19] <Evill> Maybe he doesn't like pickles.
[05:20] <bz> how else would you do it, then?
[05:20] <richardb> json.{dumps,loads}
[05:21] <Evill> I'd eat my burgers as intended, rather than removing essential ingredients.
[05:21] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[05:22] <bz> so ultimately, you're still de-/serializing
[05:23] <michel_v> this is why I don't order hamburgers that feature pickles
[05:25] <Evill> Burgers are better than cerial anyway.
[05:30] br1_ (~BrunoM@r190-135-60-64.dialup.adsl.anteldata.net.uy) joined #redis.
[05:33] lifeeth (~praneeth@unaffiliated/lifeeth) left irc: Quit: Up and at 'em, Atom Ant!
[05:34] br1 (~BrunoM@r190-135-7-142.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 276 seconds
[05:37] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[05:39] <horsey> bz: Nope, I tried json.dump() on the returned string. No use.
[05:39] <horsey> json.dumps(), rather.
[05:39] georgebashi (~george@188-223-59-50.zone14.bethere.co.uk) left irc: Quit: georgebashi
[05:42] issackelly (~issackell@cpe-98-28-23-120.columbus.res.rr.com) left irc: Quit: Computer has gone to sleep.
[05:49] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) joined #redis.
[05:50] djanowski (~djanowski@190.227.99.157) joined #redis.
[05:50] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[05:51] tesseracter (tesseract@c-66-31-28-249.hsd1.ma.comcast.net) left #redis ("Leaving").
[05:51] <bz> horsey: uh, loads translates the string back to a dict. dumps does the reverse.
[05:52] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[05:52] <bz> horsey: so, json.loads(redis.Redis().get("key_to_stringified_dict"))
[05:53] KevBurns (~KevBurnsJ@63.146.69.17) joined #redis.
[05:55] KevBurnsJr (~KevBurnsJ@75.52.124.38) left irc: Ping timeout: 250 seconds
[05:56] <horsey> bz: loads does not work either. I get "TypeError: expected string or buffer"
[05:57] <horsey> "{'count': 1, 'name': u'test-icon', 'ctime': '18:01:46 01/12/11 IST', 'mime': 'image/jpeg', 'ownerid': u'345', 'size': 112605}" <-- This is the string returned by Redis.
[06:02] KevBurns (~KevBurnsJ@63.146.69.17) left irc: Read error: Connection reset by peer
[06:03] nirvdrum (~nirvdrum@pool-173-48-56-214.bstnma.fios.verizon.net) joined #redis.
[06:08] <bz> horsey: http://pastebin.com/4MHixb4d
[06:09] KevBurnsJr (~KevBurnsJ@63.146.69.17) joined #redis.
[06:09] KevBurnsJr (~KevBurnsJ@63.146.69.17) left irc: Read error: Connection reset by peer
[06:09] KevBurnsJr (~KevBurnsJ@63.146.69.17) joined #redis.
[06:11] KevBurnsJr (~KevBurnsJ@63.146.69.17) left irc: Read error: Connection reset by peer
[06:12] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[06:14] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) joined #redis.
[06:15] KevBurnsJr (~KevBurnsJ@63.146.69.17) joined #redis.
[06:16] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Read error: Connection reset by peer
[06:17] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[06:20] jtsnow (~quassel@VDSL-151-118-130-166.DNVR.QWEST.NET) left irc: Remote host closed the connection
[06:20] KevBurnsJr (~KevBurnsJ@63.146.69.17) left irc: Read error: Connection reset by peer
[06:22] merlin83 (merlin83@unaffiliated/merlin83) left irc: Ping timeout: 240 seconds
[06:26] antirez (~antirez@host74-217-static.118-2-b.business.telecomitalia.it) joined #redis.
[06:26] #redis: mode change '+v antirez' by ChanServ!ChanServ@services.
[06:27] merlin83_ (merlin83@unaffiliated/merlin83) joined #redis.
[06:27] guido_g (~guido@nexus.a-nugget.org) left irc: Quit: Boom...
[06:32] <antirez> richardb: hi
[06:36] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[06:36] <richardb> hey
[06:36] <richardb> wanna talk about b-trees? :)
[06:39] ojwb (olly@atreus.tartarus.org) joined #redis.
[06:39] jdunck (~jdunck@75-49-157-65.lightspeed.dllstx.sbcglobal.net) joined #redis.
[06:39] <richardb> Hi ojwb.
[06:39] <ojwb> hello
[06:39] <richardb> antirez: ojwb is another Xapian developer, who's been working on the Xapian btree implementation more recently and intensively than me.
[06:40] <richardb> I expect redis will have a rather different usage pattern for its storage than Xapian does - for Xapian, it's very often sequential reads and writes.
[06:41] <richardb> but for redis it's more often going to be random access, I'd guess
[06:41] <ojwb> yeah, with some skipping over entries
[06:41] <antirez> richardb: hola
[06:41] <antirez> sorry
[06:41] <antirez> ojwb: hey
[06:41] <antirez> cool
[06:41] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Read error: Connection reset by peer
[06:41] <antirez> thank you very much
[06:41] <antirez> for this opportunity
[06:41] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[06:42] <antirez> so... I'm trying to implement a decent B-TREE for Redis :)
[06:42] <antirez> for diskstore
[06:42] br1_ -> br1
[06:42] <antirez> richardb: do you know our 'diskstore' experiment?
[06:42] <richardb> I do, but I doubt ojwb does
[06:42] <richardb> it's basically a key-value store?
[06:42] <antirez> ok, from our needs, we need on disk just a basic key-value store
[06:42] <antirez> string -> string map
[06:43] <antirez> requirements: very large datasets
[06:43] <antirez> automatic space reclaiming
[06:43] <ojwb> i know roughly what redis is - someone demoed it at perl mongers a while back, but not a lot more than that
[06:43] <antirez> and if it is possible... not too easy to corrupt :)
[06:43] djanowski (~djanowski@190.227.99.157) left irc: Remote host closed the connection
[06:43] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) left irc: Remote host closed the connection
[06:43] <antirez> ojwb: ok, but from our point of view, Redis talks to the B-TREE only via a plain key-value interface
[06:43] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[06:43] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[06:43] <richardb> The Xapian implementation satisfies the last of those, anyway.
[06:43] <antirez> so we can abstract away Redis semantic completely :)
[06:44] <richardb> You don't need to support concurrent writers, I'd guess?
[06:44] <antirez> absolutely not
[06:44] <richardb> avoiding that makes it far simpler to implement.
[06:44] <antirez> I need, optionally
[06:44] <antirez> to support reading a key that no body is writing, concurrently
[06:44] <antirez> but I can live without this
[06:44] <antirez> so I can have at amx
[06:44] <antirez> a wirter against key B
[06:44] <antirez> and a reader against key A
[06:45] <antirez> but with minor changes, I can just add a lock so that I'll serialize everything
[06:45] <ojwb> xapian's btrees essentially support a snapshot for the reader to look at
[06:45] <antirez> as the concurrent read only happens during very strange operations
[06:45] <richardb> Xapian's implementation works by having the writer only write to "new" blocks.
[06:45] <antirez> ok, that is similar to ldap's BSD btree
[06:45] <richardb> and then switches the root after an update to switch readers to reading the new version
[06:46] <richardb> you could in principle support arbitrary old revisions of the database for readers.
[06:46] <antirez> so I guess, there is a specific space compaction step
[06:46] <richardb> We keep track of which blocks are in use for each revision.
[06:46] <richardb> So we can find blocks which aren't in use for any revision
[06:46] <richardb> and when a new block is requested, one of those is used first.
[06:47] <antirez> cool
[06:47] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) joined #redis.
[06:47] <antirez> is this implementation written in C?
[06:47] <richardb> C++
[06:47] <antirez> ok, do you use mmap()?
[06:47] <richardb> no, pread/pwrite
[06:47] <richardb> ojwb: did we ever try with mmap?
[06:47] <ojwb> well, the main issue is that you are limited to the size of the address space
[06:47] <antirez> using mmap the page-cache is automatically handled by the OS, basically
[06:48] <ojwb> I tried mapping in blocks instead of pread()
[06:48] <ojwb> you're limited if you mmap the whole file that is
[06:48] <ojwb> doing it with blocks wasn't faster
[06:48] <antirez> there is this problem indeed, that in 32 bit systems can't go over 4GB
[06:48] <ojwb> i've also read that causing a page fault to load a block in is quite an overhead
[06:48] <antirez> so since you use pread/pwrite
[06:49] <antirez> yeah, it's blocking with mamp
[06:49] <antirez> mmap
[06:49] <antirez> but
[06:49] <antirez> I've an IO thread doing all the I/O
[06:49] <antirez> so Redis does not have to wait for this IO
[06:49] <antirez> what is interesting about mmap is that
[06:49] <antirez> there are completely no concerns about caching nodes in memory
[06:50] <antirez> it's completely handled by the OS
[06:50] <richardb> slow IO will cause bad latency for queries, of course.
[06:50] <antirez> yes but our assumption is that
[06:50] <richardb> We don't cache nodes in Xapian, actually.
[06:50] <antirez> the working set is likely in the redis object cache
[06:50] <antirez> ah so it's a direct access business
[06:50] <richardb> We have "cursors", which hold a set of blocks - but only one block at each level of the tree.
[06:51] fedesilva (~fedesilva@r200-40-68-22.ae-static.anteldata.net.uy) joined #redis.
[06:51] <antirez> ok, interesting
[06:51] <richardb> Which cuts out most of the re-reading of blocks.
[06:51] <antirez> and this cursors are also used for writes?
[06:51] <richardb> yes
[06:51] <ojwb> each block in the cursor has a "modified" flag in that case
[06:52] <antirez> so it will be flushed on disk at some point
[06:52] <ojwb> so it's written out when the cursor moves
[06:52] <ojwb> yes
[06:52] <antirez> how do you avoid corruption??
[06:52] <antirez> sorry for the double "?"
[06:52] <ojwb> so the upper levels of the cursor don't get written out often
[06:52] akahn (~akahn@204.145.67.146) joined #redis.
[06:52] <ojwb> well, it's a new revision
[06:52] <ojwb> not switched live until the new blocks are all written
[06:53] <richardb> So, we keep track in the cursor of whether each block has been modified
[06:53] <richardb> and when making a new revision live (ie, doing a commit), we write any dirty blocks
[06:53] <richardb> then fsync
[06:53] <antirez> ok, so your only window for problems is that
[06:53] <ojwb> but to modify a block, it's always copied to a new block first
[06:53] <antirez> the OS can re-arrange writes
[06:54] <antirez> before the fsync is complete
[06:54] <richardb> then update the pointer to the root block to point to the the block, then fsync again
[06:54] <antirez> right, with a write-barrier
[06:54] <richardb> so, that becomes slow, which is a problem
[06:54] <antirez> with mmap() there is exactly the same problem btw
[06:54] <ojwb> the kernel seems to handle fsyncs with writes between quite badly sadly
[06:54] <antirez> msync() is slow but must be used as a write-barrier
[06:54] <richardb> just there's no risk of corruption, because if we get a problem during or before the first fsync, we don't update the root, so no reader uses the potentially corrupted blocks.
[06:55] <antirez> yep got it after you mentioned write + fsync() + write
[06:55] <ojwb> well, there's a risk that fsync() doesn't actually ensure everything is on the disk platters in a permanent manner
[06:55] <antirez> this is safe but slow indeed
[06:55] <antirez> ojwb: did you experimented corruptions in the real world?
[06:55] <ojwb> fsync() doing nothing conforms to posix!
[06:55] <antirez> hehe right
[06:56] <ojwb> very few
[06:56] <ojwb> more often than not, people find there's an issue with the disk hardware
[06:56] <ojwb> if they find corruption
[06:56] <antirez> very interesting
[06:56] <ojwb> we've had some issues with the insane way revisions are updated, and that wedging
[06:57] <ojwb> but those aren't due to fsync
[06:57] <antirez> very interesting
[06:57] <antirez> so my guess is
[06:57] <antirez> mmap() can be an option but wit this big problem of size limit...
[06:57] <ojwb> currently the revison for each table is done separately, so you need to recover from only some tables being committed when you open the db
[06:57] <antirez> not very cool
[06:58] <ojwb> the new plan is to commit them together in one atomic operation
[06:58] <antirez> ok so you have some kind of journal?
[06:58] <ojwb> no, not currently
[06:58] <ojwb> it just looks for a full set of tables at the same revision
[06:58] <antirez> in theory what I need is to update multiple keys in a transactional way
[06:59] <antirez> so that if I write A and B, both or nothing will be written
[06:59] <antirez> I guess without a journal this will be hard to achieve
[06:59] <ojwb> that's easy enough with the versioned tree like xapian has
[07:00] <richardb> just don't make a new root block public until you've done both writes.
[07:00] <ojwb> http://www.google.com/search?q=Ohad+Rodeh.+"B-trees%2C+Shadowing%2C+and+Clones";
[07:00] <ojwb> that's a good paper to read
[07:00] <ojwb> it's what btrfs uses
[07:00] <antirez> great, thanks
[07:00] <antirez> but in my limited understanding of B-TREEs
[07:00] <ojwb> and is pretty similar to what xapian does, though ours isn't based on the paper
[07:00] <antirez> if two keys are in two sub-trees
[07:00] <antirez> the root node will not change after adding A and B
[07:01] <ojwb> you never modify the existing tree
[07:01] <ojwb> you build a new tree with copies of any modified blocks
[07:01] <richardb> so the root node is copied for every update
[07:01] <antirez> but isn't this against the goal of automatic-compaciton?
[07:01] <ojwb> which shares all the unmodified blocks
[07:01] <ojwb> well, you get to reuse the blocks from old revisions eventually
[07:01] <ojwb> but yes, you need the extra space for a while
[07:02] <antirez> well, "for a while" is acceptable enough I guess :)
[07:02] <richardb> It depends how big your "transactions" are how much free space is needed.
[07:02] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[07:02] <richardb> you just need enough space to hold the updates
[07:03] <antirez> well that's fair enough I think
[07:03] <antirez> it's a shame this implementation is not in C
[07:03] <antirez> what is the license?
[07:03] <ojwb> GPLv2+
[07:03] <antirez> ok, another issue ;) Well but at least I can get ideas from it
[07:04] <antirez> do you have tech documentation for your implementation?
[07:04] <richardb> There are copyright issues on the old implementation, but ojwb has full copyright in the new implementation, I think.
[07:04] <richardb> antirez: not as much documentation as we ought to have.
[07:04] <antirez> ok that's more than common, like in Redis :)
[07:04] <antirez> the .rdb file spec is "rdb.c" :)
[07:04] <ojwb> the new one lacks quite a few things currently though
[07:05] <ojwb> like releasing a block is just an empty function, so space is never reused
[07:05] <ojwb> (need to write freelist handling)
[07:05] <richardb> Yeah, that's a bit of a problem! :)
[07:05] <ojwb> it's fine for some uses
[07:05] <antirez> yes I saw many btree implementations without space reclaiming
[07:05] <antirez> with an external compaction step
[07:05] <ojwb> you can build a database, and then compact it and end up with no wasted space
[07:06] <antirez> exactly
[07:06] <richardb> Oh, one interesting thing you mentioned on twitter, antirez - you want to allow arbitrarily long keys?
[07:06] <antirez> yes, my plan was something like this
[07:06] paulsmith (~paul@c-68-55-77-184.hsd1.md.comcast.net) joined #redis.
[07:06] <antirez> in the node itself, put the space to... up to 32 bytes keys
[07:06] <antirez> but if it is longer, I'll set the first byte to something like 0xff
[07:06] <antirez> that tells you that the next 8 bytes are actually a pointer to the real key name
[07:07] <antirez> ad the next four the length of this key
[07:07] <antirez> something like that
[07:07] <antirez> so, since keys in Redis are almost always < 32 bytes
[07:07] <antirez> in many many applications
[07:07] <antirez> everything will be local to the node most of the time.
[07:08] <antirez> btw this means 32 * keys space more per every node...
[07:08] <richardb> That seems quite wasteful.
[07:08] <ojwb> the main problem I can see is that if the key length is unbounded, it's going to be hard to maintain the B-tree O() guarantees
[07:08] <antirez> yes it's a lot of space
[07:08] <richardb> Though might be a good easy way to get an implementation going quickly.
[07:08] <antirez> yes if I'll have to write the implementation myself, I think I'll start with something easy
[07:09] <richardb> Where would you store the actual long-keys?
[07:09] <ojwb> though if they are actually almost all short in practice, you may get away with it
[07:09] <antirez> what you suggest as the simplest b-tree that could be good for me?
[07:09] <antirez> richardb: I will call btree_alloc_blocK()
[07:09] <antirez> and store it there, as values
[07:09] <richardb> right
[07:09] <antirez> every time I free something, i call the btree_free function
[07:09] <antirez> and this will go into a free list
[07:09] <antirez> that is basically a simple stack
[07:09] <antirez> for power-of-two sizes
[07:10] <antirez> so I've a stack for <= 8, <= 16, ... up to 64k
[07:10] <antirez> and a stack for everything > 64k
[07:10] <antirez> that's the first idea, not sure it makes sense
[07:10] <richardb> ojwb: I don't think this scheme breaks that O() guarantees, since the data stored is outside the block
[07:10] <antirez> this stack would not be fixed size
[07:10] roidrage (~roidrage@62-220-4-146.berlikomm.net) joined #redis.
[07:10] <antirez> so if I end out of stack for <= 8 bytes fre elist
[07:11] <antirez> I allocate a new one, and set the block "next" pointer
[07:11] <antirez> this is why I was tempted by mmap() :)
[07:11] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 272 seconds
[07:11] <antirez> it would make everything so trivial
[07:11] <antirez> no caches
[07:11] <antirez> just write barriers done with msync()
[07:12] <antirez> but it's a too bad limit that 4GB issue with 32 bit systems
[07:12] <ojwb> richardb: doesn't having comparing keys being arbitrarily costily present a problem?
[07:12] <ojwb> it's rather late here, so perhaps it isn't
[07:12] <richardb> Ah - the O() lookup guarantees - yes, those get broken, quite right.
[07:12] <richardb> I was just thinking of the O() space guarantees.
[07:12] <antirez> let's say that the amortized time is ok as long as
[07:12] <antirez> the average key length is near a small number
[07:12] <ojwb> antirez: you can store the freelist in blocks which are in the freelist
[07:13] <ojwb> so it essentially takes no space
[07:13] <antirez> so I can say, this implementation is Big-O friendly IF you use small keys in the average
[07:13] <ojwb> since if there are no blocks in the freelist, there's nothing to store
[07:13] <antirez> ojwb: cool
[07:13] <antirez> lol
[07:13] <antirez> hehe I like that
[07:13] <antirez> question...
[07:13] <antirez> what about if
[07:13] <antirez> I handle large keys creating a B-tree implementation that always SHA1 keys?
[07:14] <antirez> so I always have fixed length keys
[07:14] <antirez> 20 bytes each
[07:14] roidrage (~roidrage@62-220-4-146.berlikomm.net) left irc: Client Quit
[07:14] <antirez> collisions are more or less impossible
[07:14] <antirez> and it's so fast to match this way
[07:14] <ojwb> you wouldn't get a useful ordering, if that's an issue
[07:14] <antirez> also
[07:14] <antirez> this will provide a wonderfully balanced tree
[07:14] <antirez> right, this is not an issue for Redis
[07:14] <antirez> what I need is just that I
[07:14] <antirez> can access all the keys in a given order, whatever it is
[07:15] <antirez> this is because in memory we use an hash table
[07:15] <richardb> Incidentally, there is a btree-like algorithm which allows arbitrarily long keys, and only stores a single character of each key in the non-leaf blocks (and an offset of where the character occurs). It's called a "string btree" (google finds a 1999 paper describing it), and uses "Patricia trees" inside each block. I tried implementing it once, though - nightmare!
[07:15] <antirez> so it's already unsorted :)
[07:15] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[07:15] <antirez> richardb: sonuds cool but too complex for me :)
[07:15] guido_g (~guido@nexus.a-nugget.org) joined #redis.
[07:15] <ojwb> SHA1(key) would probably work then
[07:16] <richardb> yeah - interesting read for ideas, but not a practical one to implement, I think.
[07:16] <antirez> yes I'm liking this SHA! stuff
[07:16] <ojwb> fwiw, you should also be to not store the full keys in most blocks
[07:16] <ojwb> since they'll tend to have a common prefix
[07:16] <antirez> right
[07:16] <ojwb> that's the plan for brass, but I've not quite figured out how to sanely handle inserting a key which doesn't share the prefix
[07:16] <richardb> antirez: people often use prefixes to group keys in redis, I think, and keys with a common prefix will often be accessed together.
[07:16] <ojwb> do you recalculate, or have some exception mechanism or what
[07:17] <antirez> yes, but I'm not concerned with space efficiency on disk
[07:17] <richardb> So you're likely to get better locality of reference if you don't hash the keys.
[07:17] <antirez> as long as it takes a reasonable amount of space is ok for Redis needs
[07:17] <ojwb> well, space efficiency on disk ~= speed when you're I/O bound
[07:17] <richardb> Have you looked at sqlite's backend? IIRC, that's a btree one.
[07:18] <richardb> (and not very space efficient!)
[07:18] <antirez> right, but while we need a decently fast implementation, we don't need it to be super fast
[07:18] <ojwb> http://liw.fi/btree/ is a python implementation of the Rodeh paper btrees
[07:18] <antirez> since after all it is assumed that the user is using a few gigabytes of object cache
[07:18] <antirez> so in theory, the working set, should all work in memory
[07:18] <antirez> for ojwb I need to specify a few things
[07:18] <ojwb> not read the code, but lars described it
[07:19] <antirez> when redis transfers an object on disk, this object gets encoded
[07:19] <antirez> and decoded when we read it
[07:19] <antirez> basically disk I/O is pretty costly for us anyway
[07:19] <antirez> so there is the general assumption that the user will try to avoid being I/O bound
[07:19] <antirez> everything should work well enough even if we are I/O bound, just slowly
[07:19] <antirez> but it's better to avoid this :)
[07:20] <richardb> and only those requests which are IO bound will be slowed down, of course
[07:20] <antirez> so currently I could live with a slower implementation that is very easy to imlpement
[07:20] <antirez> yes, depends a lot on the diskstore settings of course
[07:20] <antirez> the object flush delay time and so forth
[07:20] <antirez> currently I could like to have a compromise between simplicity
[07:20] <ojwb> richardb: though you'll run out of spare RAM for the OS to cache stuff in too
[07:20] <antirez> and speed
[07:21] <antirez> yep, better to avoid to be too space inevvicient
[07:21] <antirez> btw
[07:21] <richardb> Doing a hash of the keys, so that they fit into a bounded length is probably a good idea, for simplicity, then.
[07:21] <ojwb> anyway, i certainly wouldn't obsess on it
[07:21] <antirez> 20 bytes per key is near to many datasets average key length
[07:21] <richardb> handling out-of-block keys sounds tricky to get right
[07:21] <antirez> and slow
[07:21] <antirez> as there is a seek
[07:21] <richardb> for sure
[07:21] <ojwb> antirez: plus you can avoid storing the length if it is fixed
[07:21] <antirez> ojwb: good point!
[07:22] elcuervo (~elcuervo@r186-48-221-215.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 240 seconds
[07:22] <richardb> I imagine handling large values would be a common requirement.
[07:22] <antirez> btw it is probably worth doing some math
[07:22] <antirez> about collisions
[07:22] <antirez> as maybe I don't need 20 bytes of SHA1
[07:22] <antirez> but less
[07:22] <richardb> sure
[07:22] <antirez> 128 should be enough
[07:22] <ojwb> you may be able to use something that isn't aimed to be cryptographically secure, but is faster
[07:23] <richardb> Or you could handle collisions
[07:23] <antirez> yep, MD4 can be an option
[07:23] <antirez> I like the idea of not handling collisions :)
[07:23] <antirez> you can add a lot of asserts inside the code! ;) hehe
[07:23] <richardb> :) If the key is hashed, will you need to store the key at all?
[07:23] <ojwb> i think you will if you want to iterate them
[07:24] <antirez> yes I've to store the full key anyway, in the value
[07:24] <ojwb> or just write unsha1(hash)
[07:24] <antirez> ojwb: lol
[07:24] <richardb> heh
[07:24] <antirez> the reason I need the full key/value stuff
[07:24] <antirez> in the value side of the-btree
[07:24] <antirez> is that I need to implement bgsave just as a fast copy operation, for one
[07:24] <antirez> and
[07:24] <antirez> I need to be able to create an .rdb file easily starting from a corrupted b-tree
[07:25] <antirez> as data safety is very important for Redis users
[07:25] KevBurnsJr (~kevburnsj@c-98-234-48-188.hsd1.ca.comcast.net) joined #redis.
[07:25] <antirez> so if I prefix every value with a random fixed 64 bit string
[07:25] <antirez> it's possible to recover everything inside the b-tree just scanning sequentially for this string
[07:25] gnrfan (~gnrfan@190.234.136.133) joined #redis.
[07:25] <antirez> and every time I found it, I know a key-value encoded pair follow (in redis .rdb format)
[07:25] <antirez> so as bad as the crash can be, everything will be recovered most of the time
[07:26] <antirez> ok, I guess that I understood a few important things here
[07:26] <antirez> 1) that mmap() is out of question unfortunately
[07:27] <antirez> 2) that I can use SHA1 :)
[07:27] <antirez> 3) that I can use free blocks for the freelist itself
[07:27] bzinger (~bzinger@beta.blubolt.com) left irc: Quit: bzinger
[07:27] <antirez> and a few more tricks... I hope I'll not do a mess
[07:27] <ojwb> i've not implemented the free list in free blocks yet
[07:28] <ojwb> but I did convince myself it worked
[07:28] <antirez> btw, don't you think is funny that there is no BSD licensed good C btree implementation that is easy to reuse?
[07:28] <antirez> I mean, programmers write btrees since.. a lot of years :)
[07:28] <ojwb> it's slightly tricky to keep track of where you've got to in step with which revisions are live
[07:28] <antirez> And I guess the one I'm going to write will end being specific enough too perhaps
[07:29] <antirez> we should create a consortium :)
[07:29] <ojwb> i think they often are
[07:29] <antirez> and write a good btree lib :)
[07:29] <antirez> all together
[07:29] <ojwb> you probably actually want B+ trees btw
[07:30] elcuervo (~elcuervo@r186-48-214-208.dialup.adsl.anteldata.net.uy) joined #redis.
[07:30] <antirez> ojwb: yep the modification where values are always in the leaf?
[07:30] <antirez> I read about it in introduction to algorithms
[07:30] <richardb> There are variants where the leaf blocks are a different size to the internal blocks
[07:30] <richardb> usually larger
[07:31] <richardb> which can help with storing long values
[07:31] <ojwb> antirez: yes, that's the one
[07:31] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) joined #redis.
[07:31] hobodave (~hobodave@pdpc/supporter/professional/hobodave) left irc: Remote host closed the connection
[07:31] <antirez> right, I for sure need values of any size
[07:31] <richardb> Though it's probably better to just have a separate policy for values which are longer than 1/4 of the block size.
[07:31] <ojwb> there are various variants, often not referred to with consistent names
[07:31] <antirez> big lists , hashes, ... serialized will be big
[07:31] <ojwb> currently brass uses a separate store for larger values
[07:32] <antirez> interesting
[07:32] <richardb> Xapian's current database splits values into chunks, and adds a two-byte suffix to the key to store those
[07:32] <richardb> which works okay, particularly with the optimisations for sequential access
[07:32] <ojwb> but that suffix and accounting for it actually eats quite a lot of space
[07:33] <ojwb> so my idea was to get rid of that, and let that space store a pointer when we need it
[07:33] <richardb> (ie, if you want to read a value, you have to iterate over the values stored for "key\x00\x00", "key\x00\x01", until you get a key which doesn't start with "key")
[07:33] <antirez> wow, you are seriously into this
[07:33] <ojwb> it's not clear how that will work out though
[07:33] <antirez> my idea was truly naive, that is
[07:33] <antirez> to just have pointers to values directly inside nodes
[07:33] <antirez> so that values are always external
[07:33] <richardb> antirez: we've got a 10 year + headstart on you - we should have got further than we have!
[07:34] <ojwb> well, i do spend a lot of time scribbling mad ideas on bits of paper
[07:34] <antirez> hahaha :)
[07:34] <antirez> very cool btw
[07:34] <antirez> assuming my super-simple implementation can't be further complicated
[07:34] <antirez> what do you think about
[07:34] <antirez> using the same trick I wanted to use for keys?
[07:34] <antirez> so
[07:34] <antirez> I've this btree node
[07:34] <antirez> that is a mix of
[07:34] <antirez> fixed-size length
[07:34] <antirez> ekys
[07:34] <antirez> 16 bytes each
[07:34] <antirez> and that is one...
[07:35] <antirez> of course, pointers to childs
[07:35] <antirez> that will be 8 bytes each
[07:35] <ojwb> i think the key thing is taken from the bcpl muscat, so it's probably 20+ years
[07:35] <antirez> and...
[07:35] <antirez> ok I guess
[07:35] <antirez> I need some special way to flag pointers to values
[07:35] <richardb> so, in the leaf nodes, you don't have pointers to children, of course.
[07:36] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) left irc: Remote host closed the connection
[07:36] <richardb> Instead, you could have a separate file holding all the values, which you point into.
[07:36] <antirez> right, I can reuse the space to point to values
[07:36] <richardb> or you could store a block number, and allow blocks to contain values.
[07:36] <antirez> I would love to have an implementation creating a single file
[07:36] <antirez> exactly
[07:36] <antirez> but
[07:36] <antirez> what about if I don't allocate in blocks?
[07:36] <richardb> I'd definitely aim for writing a single file
[07:36] <antirez> and write something like an on-disk malloc
[07:37] <antirez> so I can btree_alloc(12):
[07:37] <antirez> the the amount I need
[07:37] <antirez> the allocation will be prefixed with the allocation length itself
[07:37] <richardb> which would presumably allocate 12 bytes in blocks, somewhere.
[07:37] <ojwb> that's kind of where I'm going for storing the larger values
[07:37] <richardb> Would you allow btree_alloc() to be passed sizes larger than the block size?
[07:37] <antirez> exactly
[07:37] <antirez> and I store the size of the block before the block
[07:37] <antirez> so when I call
[07:37] <antirez> btree_free()
[07:38] <antirez> I'll know the size of this block, and can put it into the free list
[07:38] <antirez> assuming this will work well, I can allocate everything using this system
[07:38] <richardb> You'll have definite fragmentation risk
[07:38] fbru02_ (~Federico@r186-48-221-215.dialup.adsl.anteldata.net.uy) joined #redis.
[07:38] <antirez> space for nodes, for values, for freelist blocks
[07:38] <antirez> yep either the freelist is very good or this wil not work
[07:38] <antirez> so
[07:38] <antirez> when I call btree_alloc()
[07:38] <antirez> it will try to get a block that is no longer used
[07:39] <antirez> checking the free list
[07:39] <antirez> what I don't like about this schema is how easy it is to corrupt things
[07:39] <antirez> after a crash the free list may not be marked as free
[07:39] <antirez> so I can reuse it after the reboot, for an error
[07:39] <richardb> I suspect that either you'll have to spend a lot of time on the allocator, or do periodic cleanups of the database to reduce fragmentation.
[07:40] <richardb> If you trust fsync, and do the write - fsync - write new root - fsync pattern, you should be okay.
[07:40] <richardb> barring code bugs!
[07:40] <antirez> :)
[07:40] <antirez> yep so that crash
[07:40] <antirez> will just leak
[07:40] <antirez> so I allocate, there is a free list entry, I use this, write the metadata in the freelist header
[07:40] <antirez> and fsync()
[07:40] <antirez> then I can use this block
[07:41] <antirez> if I crash now, it will just leak this block of data
[07:41] <ojwb> if you track the allocation state along with the revisioning, it shouldn't leak
[07:41] <antirez> right
[07:41] <ojwb> since the space is only really allocated when you commit the changes
[07:41] <richardb> So, don't overwrite the old freelist - make a new copy of it (or the part of it which has changed)
[07:41] <antirez> so I can have a single fsync()
[07:42] <antirez> well if this works can be a compromise
[07:42] <antirez> I mean, I'll end with a btree that is not the fastest
[07:42] <antirez> but is simple code and will try to defragment itself
[07:42] <antirez> but of course if I'll pick a best fit
[07:42] <antirez> in the free list, will be slower but less fragmented
[07:42] <ojwb> getting something that actually works is certainly a good first goal
[07:43] <antirez> if I'll use a first-fit policy, it will fragment more
[07:43] <antirez> ojwb: haha I think so
[07:43] <richardb> I guess you'd still allocate in chunks of 2**n size, though.
[07:43] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) left irc: Remote host closed the connection
[07:43] <antirez> before of a few days ago
[07:43] <richardb> which will help a lot.
[07:43] <antirez> I had zero ideas about how to model a tree on disk at all...
[07:43] <antirez> richardb: sure
[07:43] <antirez> richardb: allocating in power of two fashion will make also possible
[07:43] <antirez> to use less space for the block size, right?
[07:43] <antirez> I can just store the exponent
[07:44] <richardb> absolutely.
[07:44] <richardb> The other way to store values, whilst keeping fixed size blocks is to spread them across multiple blocks.
[07:44] <antirez> 2^n, so I can just store n in two bytes
[07:44] <ojwb> you fragment less, but you do waste space by allocate more than you want
[07:44] <antirez> exactly, it's a space/time tradeoff
[07:44] <antirez> or better
[07:44] <antirez> a space/space tradeoff ;)
[07:44] <richardb> So, a block can hold a node in the tree, or some part of a value, or part of the freelist (identified by an initial byte)
[07:44] <antirez> richardb: ah
[07:45] <antirez> all the implementations I saw so far have this idea of page indeed
[07:45] <antirez> or block that is
[07:45] <richardb> the format for a value block would be something like: length of value chunk, block number of next block (or 0 for none), value chunk
[07:45] <richardb> To read, you'd seek through the blocks.
[07:46] <antirez> right, makes sense but
[07:46] <richardb> which hopefully would be allocated near together usually
[07:46] <antirez> if you have many small values
[07:46] <antirez> you waste a full block for each?
[07:46] <richardb> For really small values (< 1/4 block size, so can fit in an item without breaking btree constraints)
[07:46] ojwb should really go to bed
[07:46] <richardb> you just store the value in the key
[07:46] <richardb> sorry for dragging you in so late, olly!
[07:46] <antirez> ojwb: thank you very much
[07:47] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) joined #redis.
[07:47] <antirez> ojwb: have a good night! and thanks again
[07:47] <antirez> richardb: ah right, there is the trick of storing the value in the key
[07:47] <richardb> for medium size values (<1/4 block to 1 block), it might be worth being able to store part of the value.
[07:47] <richardb> in a value block which is shared with other keys.
[07:48] <antirez> makes sense, but very complex indeed...
[07:48] <richardb> If you're able to allocate blocks of varying size, it's simpler.
[07:48] <antirez> then you need possibly a refcount in the block
[07:48] <antirez> I think I'll try with the power-of-two allocator trick and freelist
[07:48] <richardb> yeah, or some such overhead somewhere
[07:48] <antirez> to create a simple on-disk malloc
[07:48] <richardb> yeah, sounds like a good idea.
[07:48] <antirez> and then I'll try to model the btree on top of this system
[07:49] <richardb> yeah - you can see how important such tweaks are later.
[07:49] <antirez> and I'll check what happens... I hope to have something working in a week at max
[07:49] <richardb> Oh, one other idea from Xapian
[07:50] <richardb> We have a global revision number for each database
[07:50] <richardb> starting at 0
[07:50] <richardb> and incrementing on each commit
[07:50] <richardb> we store that in each block, at the start of it
[07:50] <richardb> so, when you read the root block, you find out what the revision number was when the root was written
[07:50] <antirez> oh, interesting
[07:50] <richardb> as you go down the tree, the revision number should always decrease (or stay the same)
[07:50] <antirez> all the blocks > than this are not valid
[07:50] <richardb> which allows you to detect corruption
[07:51] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[07:51] <antirez> right, makes sense indeed
[07:51] <br1> antirez: have you looked at the cache oblivious btree link I sent you via twitter? I feel faster than disk seek time inserts are a killer feature.
[07:51] <richardb> Or, as a reader, to detect if the version you're reading is so old that it's been discarded.
[07:51] mrflip_ (~mrflip@cpe-67-9-146-29.austin.res.rr.com) left irc: Quit: mrflip_
[07:51] <antirez> br1: it's in my collection of links to read, thanks :)
[07:52] el_kevino (~el_kevino@tor.office.avidlifemedia.com) joined #redis.
[07:52] <richardb> we also used to store the version number at the end of the block
[07:52] <richardb> and read both and check they were the same
[07:52] <richardb> as a paranoid measure to detect corruption from disk reordering
[07:52] <antirez> interesting
[07:52] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[07:52] <richardb> it never detected any, though!
[07:53] <antirez> cool :)
[07:53] <antirez> so fsync() is often doing its work
[07:53] <richardb> though disks used to be dumber and less likely to reorder behind your back!
[07:53] <antirez> right...
[07:53] <richardb> The nice thing about Xapian's implementation is that you can pull the plug at any time, and when you reboot you can just continue using the database - no rollback/recovery step to do.
[07:54] <richardb> antirez: I assume you've not started coding on this yet?
[07:54] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[07:54] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[07:54] <antirez> richardb: not a line of code at all. Just a design document with random ideas
[07:55] abecc (~abecc@135-159-126-200.fibertel.com.ar) joined #redis.
[07:55] <richardb> I can't promise time to help (I shouldn't really be here today - backlog of stuff to do is quite huge), but this is a really fun topic for me, so I'd definitely like to give as much as I can find.
[07:56] <richardb> and would definitely like to see a nice, reasonably generic, BSD (or looser!) licensed btree implementation appear.
[07:56] <antirez> richardb: thank you, so I'll try to ping you when I've something :)
[07:56] <antirez> richardb: in the hope you can do a five minutes check
[07:56] <richardb> sure, do!
[07:56] <antirez> thank you, this was super useful.
[07:57] <antirez> please can you give me also the twitter nick of ojwb?
[07:57] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[07:57] <richardb> Glad Olly could make it too.
[07:57] <antirez> so that I can publicly thank you both?
[07:58] felixge (~felixgeis@p579F8BA0.dip.t-dialin.net) joined #redis.
[07:58] felixge (~felixgeis@p579F8BA0.dip.t-dialin.net) left irc: Changing host
[07:58] felixge (~felixgeis@miranda/donor/theundefined) joined #redis.
[07:58] <richardb> good question - I'm not sure he was a twitter a/c
[07:59] <antirez> ok :)
[07:59] <antirez> I'll use the IRC nick
[07:59] <richardb> He's on identi.ca as olly!
[07:59] <antirez> ah ok
[07:59] <richardb> and @xapian gets tweets from him
[08:00] bzinger (~bzinger@beta.blubolt.com) joined #redis.
[08:00] zevarito_ (~zevarito@r186-48-221-215.dialup.adsl.anteldata.net.uy) joined #redis.
[08:01] <antirez> richardb: great, thank you! away from keyboard for a bit, then back here. Later!
[08:01] <zevarito_> There is posible to have a key which points to another key, like an alias ?
[08:01] <richardb> cheers
[08:03] Sarevok (~locke@cpe-065-184-240-227.ec.res.rr.com) joined #redis.
[08:03] hobodave (~hobodave@pdpc/supporter/professional/hobodave) joined #redis.
[08:09] seivan (~seivan@109.228.150.197) joined #redis.
[08:10] andymccurdy (~andymccur@c-67-188-242-100.hsd1.ca.comcast.net) left irc: Quit: Computer has gone to sleep.
[08:12] abecc (~abecc@135-159-126-200.fibertel.com.ar) left irc: Quit: abecc
[08:15] <nirvdrum> Is there a way for me to see what changes are pending writes to disk? I have an absurdly high number of modifications occurring in a very short period of time, which is indicative of a bug in my app, but I'm not sure where to start looking.
[08:15] djanowski_ (~djanowski@190.227.83.112) joined #redis.
[08:16] #redis: mode change '+v djanowski_' by ChanServ!ChanServ@services.
[08:16] <nirvdrum> If I could see what keys are constantly being modified, I could narrow my search space considerably.
[08:18] hikeonpast (~hikeonpas@71-95-209-242.static.mtpk.ca.charter.com) joined #redis.
[08:18] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Ping timeout: 276 seconds
[08:22] thinkingpotato_ (~thinkingp@abbo141.neoplus.adsl.tpnet.pl) joined #redis.
[08:24] thinkingpotato (~thinkingp@abbn74.neoplus.adsl.tpnet.pl) left irc: Ping timeout: 246 seconds
[08:24] thinkingpotato_ -> thinkingpotato
[08:24] merlin83_ -> merlin83
[08:25] danlucraft1 (~Adium@79.173.154.114) left irc: Quit: Leaving.
[08:26] pietern (~pieter@nhit.xs4all.nl) joined #redis.
[08:26] #redis: mode change '+v pietern' by ChanServ!ChanServ@services.
[08:26] danlucraft (~Adium@79.173.154.114) joined #redis.
[08:27] <pietern> good afternoon all
[08:28] danlucraft (~Adium@79.173.154.114) left irc: Read error: Connection reset by peer
[08:29] <soveran> hey pietern
[08:29] <pietern> hi soveran
[08:30] <pietern> soveran: I applied this patch yesterday to fix redis-io from throwing errors: https://github.com/antirez/redis-io/commit/25d0d56ffa42e2ae03e2909394f82c3fd3e3f37a
[08:30] <pietern> it's pretty dirty, but couldn't think of a better way to do it quickly
[08:30] juangiordana_ (~quassel@190.136.149.23) joined #redis.
[08:31] juangiordana (~quassel@190.31.86.179) left irc: Ping timeout: 265 seconds
[08:34] sechrist (~sechrist@c-76-102-99-133.hsd1.ca.comcast.net) left irc: Ping timeout: 264 seconds
[08:35] mrflip_ (~mrflip@rrcs-71-42-138-16.sw.biz.rr.com) joined #redis.
[08:38] danlucraft (~Adium@79.173.154.114) joined #redis.
[08:39] jdmaturen (~jdmaturen@c-67-160-216-89.hsd1.ca.comcast.net) joined #redis.
[08:40] <soveran> pietern: I'm fine with that
[08:40] <pietern> soveran: cool
[08:40] <pietern> soveran: btw, don't know which component is responsible for clearing the ivars
[08:40] <pietern> would that be rum?
[08:41] <soveran> pietern: maybe rack?
[08:41] <pietern> could be, very weird anyway
[08:41] <soveran> pietern: what was the error?
[08:42] <pietern> oh, the number of fds ran out
[08:42] <pietern> because for every request a new redis connection was made
[08:42] <pietern> and that was the case because the instance variable @redis on Kernel was reset on every request
[08:45] <soveran> gotcha
[08:45] gllvr (~gllvr@38.99.63.41) joined #redis.
[08:49] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) joined #redis.
[08:52] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) left irc: Ping timeout: 260 seconds
[08:56] elcuervo (~elcuervo@r186-48-214-208.dialup.adsl.anteldata.net.uy) left irc: Read error: Connection reset by peer
[08:56] elcuervo (~elcuervo@r186-48-214-208.dialup.adsl.anteldata.net.uy) joined #redis.
[09:00] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[09:02] sechrist (~sechrist@64-79-127-110.static.wiline.com) joined #redis.
[09:03] jdmaturen (~jdmaturen@c-67-160-216-89.hsd1.ca.comcast.net) left irc: Quit: jdmaturen
[09:03] fzzbt (fuzzybyte@77.79.7.8) left #redis.
[09:04] djanowski_ (~djanowski@190.227.83.112) left irc: Ping timeout: 265 seconds
[09:05] unbracketed (~brian@cpe-76-169-191-136.socal.res.rr.com) joined #redis.
[09:06] Tyler-- (~bob@76.177.102.69) left irc: Read error: Connection reset by peer
[09:06] Tyler-- (~bob@76.177.102.69) joined #redis.
[09:12] Wombert (~Wombert@zentrale.gutefrage.net) left irc: Quit: Wombert
[09:13] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) left irc: Ping timeout: 240 seconds
[09:13] mattly (~mattly@97-120-38-31.ptld.qwest.net) joined #redis.
[09:21] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) joined #redis.
[09:26] jdmaturen (~jdmaturen@204.16.157.170) joined #redis.
[09:26] jdmaturen (~jdmaturen@204.16.157.170) left irc: Remote host closed the connection
[09:27] jdmaturen (~jdmaturen@204.16.157.170) joined #redis.
[09:29] pietern (~pieter@nhit.xs4all.nl) left irc: Quit: pietern
[09:32] petercooper (~petercoop@2.97.188.45) joined #redis.
[09:35] Eugene_ (d58a56d0@gateway/web/freenode/ip.213.138.86.208) joined #redis.
[09:38] scottsc (~scottsc@204.14.156.126) joined #redis.
[09:38] scottsc (~scottsc@204.14.156.126) left irc: Remote host closed the connection
[09:41] <Eugene_> hello. I have a question about redis upgrade 2.0.0 => 2.0.4. I installed 2.0.4, configured it to be a slave of 2.0.0 and started it on different port. SYNC went fine, but for some reason new redis process constantly grows while 2.0.0 doesn't. When I looked at stats via INFO I found out that total number of keys was constantly increasing while in 2.0.0 it was almost at the same level (because most of them were supposed to expire soon)
[09:41] <Eugene_> can anybody help me to understand what's happening?
[09:42] mattly (~mattly@97-120-38-31.ptld.qwest.net) left irc: Remote host closed the connection
[09:44] watsonian (~watsonian@enginey-9.border1.sfo002.pnap.net) joined #redis.
[09:49] steffkes (~steffkes@mnch-5d874bb3.pool.mediaWays.net) joined #redis.
[09:49] Eugene_ (d58a56d0@gateway/web/freenode/ip.213.138.86.208) left irc: Quit: Page closed
[09:50] scottsc (~scottsc@vpn01.beboinc.com) joined #redis.
[09:50] scottsc (~scottsc@vpn01.beboinc.com) left irc: Client Quit
[09:53] kost_ (~kost@217.172.23.200) joined #redis.
[09:54] zevarito_ (~zevarito@r186-48-221-215.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[09:55] zevarito (~zevarito@r186-48-214-208.dialup.adsl.anteldata.net.uy) joined #redis.
[09:55] jdunck (~jdunck@75-49-157-65.lightspeed.dllstx.sbcglobal.net) left irc: Quit: jdunck
[09:56] ronr_ (~ronr@95.35.21.145) left irc: Ping timeout: 240 seconds
[09:59] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[10:00] ntelford (~ntelford@81.168.58.46) left irc: Quit: Ex-Chat
[10:02] unbracketed1 (~brian@cpe-76-169-191-136.socal.res.rr.com) joined #redis.
[10:02] Jessica_ (46599c69@gateway/web/freenode/ip.70.89.156.105) joined #redis.
[10:02] unbracketed (~brian@cpe-76-169-191-136.socal.res.rr.com) left irc: Disconnected by services
[10:03] unbracketed1 -> unbracketed
[10:03] sechrist (~sechrist@64-79-127-110.static.wiline.com) left irc: Remote host closed the connection
[10:04] <Jessica_> hi, does anybody know how to make redis server not dump the dump.rdb file, period? disabling save just doesn't save it periodically, but seems like when the server is restarted the dump file is still created
[10:08] <antirez> Jessica_: hi
[10:08] <antirez> Jessica_: are you using replication?
[10:08] <antirez> oh wait, I know what it is
[10:09] <antirez> Jessica_: switch to Redis 2.2 (anyway a good idea), and the problem will disappear
[10:09] <antirez> in Redis 2.0
[10:09] <antirez> there is a known bug, when you kill the server it will dump
[10:09] <antirez> regardless of the fact it's configured for not saving
[10:09] <antirez> 2.2 got it right
[10:09] <Jessica_> ah, ok, easy enough :)
[10:09] <Jessica_> thanks!
[10:10] <antirez> Jessica_: are you using Redis as a cache?
[10:10] <antirez> because, redis 2.2 got some interesting new features about it
[10:10] akahn (akahn@204.145.67.146) left #redis.
[10:10] lifeeth (~praneeth@202.3.77.210) joined #redis.
[10:10] lifeeth (~praneeth@202.3.77.210) left irc: Changing host
[10:10] lifeeth (~praneeth@unaffiliated/lifeeth) joined #redis.
[10:11] <Jessica_> yes, i am using redis as a cache
[10:12] <antirez> Jessica_: in Redis 2.2 you have new parameters for the maxmemory setting
[10:12] <antirez> so that you can select: maxmemory-policy allkeys-lru
[10:12] <antirez> when the memory limit is reached
[10:12] <Jessica_> ooooh
[10:12] <antirez> any key will be evicted, without any need to set an EXPIRE
[10:12] <Jessica_> cool
[10:12] <antirez> memcached-alike
[10:13] <Jessica_> cool, i'll definitely look into that :) is 2.2 safe for prod?
[10:13] mjr_ (~mjr@c-67-170-192-232.hsd1.ca.comcast.net) left irc: Quit: mjr_
[10:13] <antirez> Jessica_: for a cache definitely yes, but in general, I suggest people switching over 2.2 ASAP
[10:14] <Jessica_> sgtm, thanks for your help!
[10:14] <antirez> you are welcome
[10:14] andymccurdy (~andymccur@69.12.160.66) joined #redis.
[10:15] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 272 seconds
[10:17] runa (~asd@190.226.110.89) joined #redis.
[10:18] robstorrs (~robstorrs@204.16.157.170) joined #redis.
[10:19] <runa> heyas. I have redis configured with vm-enabled and vm-max-memory but it still uses a lot of ram. any hints? here's the output of 'info' http://pastebin.com/uTPK41PE
[10:19] <runa> erm. vm-max-memory = 0
[10:20] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[10:20] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[10:20] <runa> djanowski: aloha
[10:20] <antirez> runa: unfortunately Redis Vm is not able to swap keys. But we are writing a new version of Redis that can do this
[10:20] nff (~nicolas@smtp-ox.owlient.eu) left irc: Quit: Leaving
[10:20] <antirez> runa: if in your project data is not very important, you can try our alpha
[10:20] sudoer (~jtoy@c-98-210-164-172.hsd1.ca.comcast.net) joined #redis.
[10:21] <antirez> runa: another thing you can do is upgrading to Redis 2.2 ASAP that uses less memory
[10:21] <antirez> runa: please describe your data set, what you have inside keys?
[10:21] <runa> antirez: some ruby Marshalled objects, data is definitely not important
[10:21] <runa> I can try the alpha?
[10:21] <runa> erm. without the "?"
[10:21] <antirez> runa: wonderful, you can definitely try the alpha :)
[10:22] <antirez> runa: simply download the 'unstable' branch from github
[10:22] <antirez> runa: you'll see that in redis.conf there is no mention of VM
[10:22] <antirez> but there is a new option called 'diskstore'
[10:22] <antirez> set diskstore-enabled to yes
[10:22] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) joined #redis.
[10:22] <runa> antirez: letme see
[10:22] <antirez> runa: ok I'll be here to help in the next 30 minutes
[10:23] bzinger (~bzinger@beta.blubolt.com) left irc: Quit: bzinger
[10:24] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[10:24] <runa> antirez: um. I have a newer version of my code that uses less keys and more Hashes; maybe I can test this first and later upgrade to the redis alpha if that didn't work
[10:24] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[10:24] <antirez> runa: yes this is very interesting, as Hashes with a few fields will use more or less the same memory of a single key
[10:25] <antirez> so if you had key a,b,c,d,e and now you have key foo -> hahs(a,b,c,d,e)
[10:25] <antirez> it's almost 5 times a memory saving
[10:25] <antirez> it's definitely the way to go for you I guess
[10:25] <runa> ok. letme see
[10:25] fedesilva (~fedesilva@r200-40-68-22.ae-static.anteldata.net.uy) left irc: Read error: Connection reset by peer
[10:26] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) left irc: Client Quit
[10:26] fedesilva (~fedesilva@r200-40-68-22.ae-static.anteldata.net.uy) joined #redis.
[10:29] <jdmaturen> antirez: re 32-bit, I imagine there are quite a few people using VMs that are 32-bit. However, how many of them have anywhere near 4GB of data?
[10:30] danlucraft (~Adium@79.173.154.114) left irc: Quit: Leaving.
[10:30] <antirez> jdmaturen: yes... good point. mmap() could seriously make the implementation simpler to write.
[10:31] <jdmaturen> simplicity is a virtue
[10:31] <antirez> indeed :)
[10:37] thinkingpotato (~thinkingp@abbo141.neoplus.adsl.tpnet.pl) left irc: Quit: Puff!
[10:40] acrylicist (~acrylicis@unaffiliated/acrylicist) joined #redis.
[10:40] mjr_ (~mjr@vpn.rebelvox.com) joined #redis.
[10:43] hikeonpast (~hikeonpas@71-95-209-242.static.mtpk.ca.charter.com) left irc: Ping timeout: 264 seconds
[10:46] o_o (~o_o@c-76-102-198-5.hsd1.ca.comcast.net) joined #redis.
[10:46] o_o -> Guest17913
[10:47] curtisc (~curtisc@75-144-19-1-ca.sfba.hfc.comcastbusiness.net) joined #redis.
[10:50] <richardb> If you wrote it with mmap, and then found that you needed to support 32-bit >4GB for some reason, you could implement a caching layer and slot it in, I think.
[10:51] <richardb> Might be awkward, but I think it would be possible, and you probably wouldn't need to do it.
[10:51] <richardb> My local development machine is 32-bit, FWIW
[10:51] <richardb> but all my deployments and public machines are now 64 bit
[10:51] richardb goes afk again
[10:54] adamholt -> adamholt_away
[10:54] adamholt_away -> adamholt
[10:56] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[11:01] noah_ (~noah@70-89-156-105-Oakley.hfc.comcastbusiness.net) joined #redis.
[11:01] antirez (~antirez@host74-217-static.118-2-b.business.telecomitalia.it) left irc: Quit: antirez
[11:07] pablohof (~prh@r190-135-35-168.dialup.adsl.anteldata.net.uy) joined #redis.
[11:08] xpressnoodles (~zaki@cpe-72-230-103-148.twcny.res.rr.com) joined #redis.
[11:09] dspezia (~dspezia@bar06-2-87-91-60-98.dsl.club-internet.fr) joined #redis.
[11:12] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[11:13] tmacedo (~tmacedo@90.163.57.2) left irc: Remote host closed the connection
[11:13] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) left irc: Ping timeout: 264 seconds
[11:14] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Ping timeout: 246 seconds
[11:15] traceback0 (~traceback@c-67-180-79-10.hsd1.ca.comcast.net) joined #redis.
[11:15] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Ping timeout: 240 seconds
[11:16] djanowski (~djanowski@186.111.109.16) joined #redis.
[11:16] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[11:16] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[11:17] fbru02_ (~Federico@r186-48-221-215.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[11:18] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 255 seconds
[11:21] Guest17913 -> robotarmy
[11:21] robotarmy (~o_o@c-76-102-198-5.hsd1.ca.comcast.net) left irc: Changing host
[11:21] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) joined #redis.
[11:24] djanowski_ (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[11:24] #redis: mode change '+v djanowski_' by ChanServ!ChanServ@services.
[11:27] djanowski (~djanowski@186.111.109.16) left irc: Ping timeout: 276 seconds
[11:28] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) left irc: Ping timeout: 240 seconds
[11:29] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Ping timeout: 240 seconds
[11:30] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) joined #redis.
[11:30] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) joined #redis.
[11:32] djanowski_ (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Read error: Connection reset by peer
[11:32] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[11:32] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[11:35] ngerakines (~ngerakine@198.74.38.57) joined #redis.
[11:37] derka (~derka@41.208.137.10) joined #redis.
[11:39] brianmario (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) joined #redis.
[11:40] derka (~derka@41.208.137.10) left irc: Client Quit
[11:40] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) left irc: Quit: Bye!
[11:41] runa (~asd@190.226.110.89) left irc: Ping timeout: 255 seconds
[11:41] Jessica_ (46599c69@gateway/web/freenode/ip.70.89.156.105) left irc: Quit: Page closed
[11:43] mrflip_ (~mrflip@rrcs-71-42-138-16.sw.biz.rr.com) left irc: Quit: mrflip_
[11:44] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[11:47] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) joined #redis.
[11:49] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[11:50] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) joined #redis.
[11:50] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[11:51] senderista (~senderist@216.161.248.54) joined #redis.
[11:52] bmizerany (~bmizerany@204.14.152.118) joined #redis.
[11:54] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[11:56] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) joined #redis.
[12:00] mrflip_ (~mrflip@rrcs-71-42-138-16.sw.biz.rr.com) joined #redis.
[12:00] namelessnotion_ (~anthony@c-68-42-68-217.hsd1.mi.comcast.net) joined #redis.
[12:01] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 260 seconds
[12:01] acts_as (~acts_as@208.236.105.27) joined #redis.
[12:02] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[12:03] namelessnotion (~anthony@c-68-42-68-217.hsd1.mi.comcast.net) left irc: Ping timeout: 240 seconds
[12:03] namelessnotion_ -> namelessnotion
[12:03] pablohof (prh@r190-135-35-168.dialup.adsl.anteldata.net.uy) left #redis.
[12:16] brianmario_ (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) joined #redis.
[12:17] brianmario__ (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) joined #redis.
[12:18] seppo00101 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[12:18] jdunck (~jdunck@64.134.149.234) joined #redis.
[12:19] Wombert (~Wombert@dslb-188-099-140-107.pools.arcor-ip.net) joined #redis.
[12:20] brianmario (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) left irc: Ping timeout: 255 seconds
[12:20] brianmario__ -> brianmario
[12:20] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Read error: Connection reset by peer
[12:20] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[12:20] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Ping timeout: 264 seconds
[12:21] brianmario_ (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) left irc: Ping timeout: 240 seconds
[12:21] marcostoledo (~marcostol@95.61.110.123) left irc: Ping timeout: 260 seconds
[12:22] runa (~asd@190.225.87.123) joined #redis.
[12:24] KevBurnsJr (~kevburnsj@c-98-234-48-188.hsd1.ca.comcast.net) left irc:
[12:26] namelessnotion (~anthony@c-68-42-68-217.hsd1.mi.comcast.net) left irc: Remote host closed the connection
[12:26] namelessnotion (~anthony@c-68-42-68-217.hsd1.mi.comcast.net) joined #redis.
[12:27] roidrage (~roidrage@88.130.189.182) joined #redis.
[12:27] Evious (~n_a@s64-180-62-209.bc.hsia.telus.net) joined #redis.
[12:27] seppo00101 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Quit: Leaving.
[12:27] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[12:28] lifeeth (~praneeth@unaffiliated/lifeeth) left irc: Ping timeout: 240 seconds
[12:29] marcostoledo (~marcostol@77.209.163.75) joined #redis.
[12:33] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 255 seconds
[12:34] deepthawtz (~deepthawt@c-24-4-229-52.hsd1.ca.comcast.net) joined #redis.
[12:44] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[12:46] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) left irc: Read error: Connection reset by peer
[12:48] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) joined #redis.
[12:50] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 276 seconds
[12:51] marcosto_ (~marcostol@95.61.110.123) joined #redis.
[12:53] marcostoledo (~marcostol@77.209.163.75) left irc: Ping timeout: 246 seconds
[12:55] br1 (~BrunoM@r190-135-60-64.dialup.adsl.anteldata.net.uy) left irc: Quit: Leaving
[12:56] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[12:57] hikeonpast (~hikeonpas@71-95-209-242.static.mtpk.ca.charter.com) joined #redis.
[13:05] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 264 seconds
[13:12] fbru02 (~Federico@r190-133-135-229.dialup.adsl.anteldata.net.uy) joined #redis.
[13:17] DubLo7 (~Adium@71-83-3-210.static.aldl.mi.charter.com) left irc: Quit: Leaving.
[13:17] sudoer (~jtoy@c-98-210-164-172.hsd1.ca.comcast.net) left irc: Quit: sudoer
[13:19] tilgovi (~quassel@2001:550:c00:8:222:19ff:fe0b:8e39) joined #redis.
[13:22] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) joined #redis.
[13:22] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Read error: Connection reset by peer
[13:23] runa (~asd@190.225.87.123) left irc: Quit: User abort with /quit
[13:23] abecc (~abecc@135-159-126-200.fibertel.com.ar) joined #redis.
[13:23] soveran (~soveran@200-42-23-2.dup.prima.net.ar) joined #redis.
[13:23] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) left irc: Quit: Bye!
[13:28] elcuervo (~elcuervo@r186-48-214-208.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[13:29] yawniek (~yannick@84-72-19-136.dclient.hispeed.ch) joined #redis.
[13:30] altamic (~altamic@93-39-53-225.ip74.fastwebnet.it) left irc: Ping timeout: 240 seconds
[13:30] soveran (~soveran@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[13:30] djanowski (~djanowski@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[13:34] tg (irc@tgbit.net) left irc: Read error: Connection reset by peer
[13:34] tg (irc@tgbit.net) joined #redis.
[13:45] nirvdrum -> nirvdrum[shoveli
[13:52] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) joined #redis.
[13:58] steffkes (~steffkes@mnch-5d874bb3.pool.mediaWays.net) left irc: Quit: Verlassend
[14:01] zevarito (~zevarito@r186-48-214-208.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[14:11] rgl (~Rui@a89-154-33-145.cpe.netcabo.pt) joined #redis.
[14:17] marcosto_ (~marcostol@95.61.110.123) left irc: Quit: Computer has gone to sleep.
[14:17] roidrage (~roidrage@88.130.189.182) left irc: Quit: roidrage
[14:20] _rgl (~Rui@a89-154-33-145.cpe.netcabo.pt) joined #redis.
[14:21] rgl (~Rui@a89-154-33-145.cpe.netcabo.pt) left irc: Disconnected by services
[14:21] _rgl -> rgl
[14:23] jdunck (~jdunck@64.134.149.234) left irc: Quit: jdunck
[14:25] petercooper (~petercoop@2.97.188.45) left irc: Read error: Operation timed out
[14:27] d0k (~d0k@aef.wh.uni-dortmund.de) left irc: Quit: he does that
[14:32] drbobbeaty (~drbobbeat@38.98.137.29) left irc: Quit: drbobbeaty
[14:34] sudoer (~jtoy@c-71-202-44-24.hsd1.ca.comcast.net) joined #redis.
[14:35] issackelly (~issackell@d60-65-125-220.col.wideopenwest.com) left irc: Quit: Computer has gone to sleep.
[14:37] petercooper (~petercoop@2.97.188.45) joined #redis.
[14:47] elcuervo (~elcuervo@r190-135-9-23.dialup.adsl.anteldata.net.uy) joined #redis.
[14:48] Sarevok (locke@cpe-065-184-240-227.ec.res.rr.com) left #redis.
[14:48] felixge (~felixgeis@miranda/donor/theundefined) left irc: Quit: felixge
[14:49] paulsmith (~paul@c-68-55-77-184.hsd1.md.comcast.net) left irc: Quit: leaving
[14:53] el_kevino (~el_kevino@tor.office.avidlifemedia.com) left irc: Remote host closed the connection
[14:54] _mythz (~demis_bel@94-193-33-5.zone7.bethere.co.uk) joined #redis.
[14:57] soveran (~soveran@186.19.214.247) joined #redis.
[15:01] soveran (~soveran@186.19.214.247) left irc: Remote host closed the connection
[15:07] soveran (~soveran@186.19.214.247) joined #redis.
[15:07] DubLo7 (~Adium@32.141.49.207) joined #redis.
[15:08] matflores (~matias@190.50.97.110) left irc: Quit: Ex-Chat
[15:16] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) joined #redis.
[15:19] elliottcable (~ec@ec2-174-129-205-205.compute-1.amazonaws.com) left irc: Quit: RUMORS OF MY SURVIVAL HAVE BEEN GREATLY EXAGGERATED
[15:19] sjl (~sjl@dwaiter.rit.edu) joined #redis.
[15:24] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) left irc: Quit: brandonleach
[15:24] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) joined #redis.
[15:25] hill_ (6fc1be18@gateway/web/freenode/ip.111.193.190.24) joined #redis.
[15:26] elliottcable (~ec@ec2-174-129-205-205.compute-1.amazonaws.com) joined #redis.
[15:28] blueadept (~blueadept@cpe-24-160-96-254.tampabay.res.rr.com) joined #redis.
[15:30] hill_ (6fc1be18@gateway/web/freenode/ip.111.193.190.24) left irc: Ping timeout: 265 seconds
[15:36] abecc (~abecc@135-159-126-200.fibertel.com.ar) left irc: Quit: abecc
[15:37] deepthawtz (~deepthawt@c-24-4-229-52.hsd1.ca.comcast.net) left irc: Quit: Leaving...
[15:40] _mythz (demis_bel@94-193-33-5.zone7.bethere.co.uk) left #redis.
[15:47] dipser (~dipser@p54844350.dip.t-dialin.net) joined #redis.
[15:58] elcuervo (~elcuervo@r190-135-9-23.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 265 seconds
[15:59] adamholt -> adamholt_away
[15:59] adamholt_away -> adamholt
[16:04] Sunray (~noop@unaffiliated/sunray) left irc: Quit: You sunk my battleship!
[16:05] elcuervo (~elcuervo@r190-135-54-183.dialup.adsl.anteldata.net.uy) joined #redis.
[16:06] fedesilva (~fedesilva@r200-40-68-22.ae-static.anteldata.net.uy) left irc: Quit: zZZzzZZzz
[16:09] DubLo7 (~Adium@32.141.49.207) left irc: Quit: Leaving.
[16:13] issackelly (~issackell@cpe-98-28-23-120.columbus.res.rr.com) joined #redis.
[16:14] brandonleach (~brandonle@242.sub-69-99-133.myvzw.com) left irc: Ping timeout: 255 seconds
[16:14] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Quit: Leaving.
[16:31] dipser_ (~dipser@p548457D4.dip.t-dialin.net) joined #redis.
[16:32] dipser_ (~dipser@p548457D4.dip.t-dialin.net) left irc: Client Quit
[16:33] dipser (~dipser@p54844350.dip.t-dialin.net) left irc: Ping timeout: 240 seconds
[16:33] jdunck (~jdunck@cpe-72-190-124-161.tx.res.rr.com) joined #redis.
[16:35] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) joined #redis.
[16:36] ank (~ank@c-24-3-189-120.hsd1.pa.comcast.net) left irc: Quit: ank
[16:37] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) left irc: Read error: Connection reset by peer
[16:41] sjl (~sjl@dwaiter.rit.edu) left irc: Quit: sjl
[16:42] <ferric> wasn't there a startup or project doing on demand redis hosting?
[16:42] <ferric> ah, redistogo.com
[16:44] soveran (~soveran@186.19.214.247) left irc: Remote host closed the connection
[16:44] kost_ (~kost@217.172.23.200) left irc: Remote host closed the connection
[16:53] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) left irc: Ping timeout: 255 seconds
[16:56] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) joined #redis.
[16:56] brandonleach (~brandonle@128.sub-75-210-86.myvzw.com) joined #redis.
[17:00] xpressnoodles (~zaki@cpe-72-230-103-148.twcny.res.rr.com) left irc: Ping timeout: 272 seconds
[17:01] brianmario (~brianmari@c-76-21-12-214.hsd1.ca.comcast.net) left irc: Quit: brianmario
[17:03] brandonleach (~brandonle@128.sub-75-210-86.myvzw.com) left irc: Ping timeout: 255 seconds
[17:06] xpressnoodles (~zaki@cpe-72-230-103-148.twcny.res.rr.com) joined #redis.
[17:07] deepthawtz (~deepthawt@c-24-4-229-52.hsd1.ca.comcast.net) joined #redis.
[17:09] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) left irc: Quit: qrush
[17:09] brandonleach (~brandonle@216.sub-75-208-208.myvzw.com) joined #redis.
[17:10] soveran (~soveran@186.19.214.247) joined #redis.
[17:10] zevarito (~zevarito@r186-48-131-245.dialup.adsl.anteldata.net.uy) joined #redis.
[17:18] senderista (~senderist@216.161.248.54) left irc: Quit: senderista
[17:21] dspezia (~dspezia@bar06-2-87-91-60-98.dsl.club-internet.fr) left irc: Remote host closed the connection
[17:25] rgl (~Rui@a89-154-33-145.cpe.netcabo.pt) left irc: Ping timeout: 260 seconds
[17:27] fbru02 (~Federico@r190-133-135-229.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 260 seconds
[17:30] fbru02 (~Federico@r190-64-212-78.dialup.adsl.anteldata.net.uy) joined #redis.
[17:36] nirvdrum[shoveli -> nirvdrum
[17:37] soveran (~soveran@186.19.214.247) left irc: Remote host closed the connection
[17:41] DubLo7 (~Adium@24-247-75-139.dhcp.trcy.mi.charter.com) joined #redis.
[17:43] mrflip_ (~mrflip@rrcs-71-42-138-16.sw.biz.rr.com) left irc: Quit: mrflip_
[17:45] watsonian (~watsonian@enginey-9.border1.sfo002.pnap.net) left irc: Quit: Bye!
[17:46] acts_as (~acts_as@208.236.105.27) left irc: Quit: acts_as
[17:47] kuhrt (~kuhrt@modemcable079.149-177-173.mc.videotron.ca) left irc: Quit: Bye!
[17:48] zevarito (~zevarito@r186-48-131-245.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[17:49] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) joined #redis.
[17:58] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) joined #redis.
[17:59] robstorrs (~robstorrs@204.16.157.170) left irc: Quit: robstorrs
[18:06] mjr_ (~mjr@vpn.rebelvox.com) left irc: Quit: mjr_
[18:09] sudoer (~jtoy@c-71-202-44-24.hsd1.ca.comcast.net) left irc: Quit: sudoer
[18:20] petercooper (~petercoop@2.97.188.45) left irc: Read error: Connection reset by peer
[18:21] acts_as (~acts_as@cpe-76-87-42-32.socal.res.rr.com) joined #redis.
[18:21] petercooper (~petercoop@2.97.188.45) joined #redis.
[18:23] fedesilva (~fedesilva@173-5.dedicado.com.uy) joined #redis.
[18:27] tangfl (3d8798c2@gateway/web/freenode/ip.61.135.152.194) joined #redis.
[18:27] <tangfl> we want to store our tens of billions countes in redis
[18:28] <tangfl> but redis treat all keys and values as string, this cost too much ext memory
[18:28] <tangfl> any good ideas ?
[18:29] andymccurdy (~andymccur@69.12.160.66) left irc: Ping timeout: 255 seconds
[18:33] tangfl (3d8798c2@gateway/web/freenode/ip.61.135.152.194) left irc: Quit: Page closed
[18:35] jdunck (~jdunck@cpe-72-190-124-161.tx.res.rr.com) left irc: Remote host closed the connection
[18:35] jdunck (~jdunck@cpe-72-190-124-161.tx.res.rr.com) joined #redis.
[18:38] MattDiPasquale (~mattdipas@ool-44c7e57a.dyn.optonline.net) left irc: Remote host closed the connection
[18:39] mjr_ (~mjr@c-67-170-192-232.hsd1.ca.comcast.net) joined #redis.
[18:39] djanowski (~djanowski@190.245.30.40) joined #redis.
[18:39] #redis: mode change '+v djanowski' by ChanServ!ChanServ@services.
[18:40] djanowski (~djanowski@190.245.30.40) left irc: Remote host closed the connection
[18:46] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) joined #redis.
[18:47] elcuervo (~elcuervo@r190-135-54-183.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[18:48] mrflip_ (~mrflip@cpe-67-9-146-29.austin.res.rr.com) joined #redis.
[18:48] seppo0010 (~Adium@156-145-231-201.fibertel.com.ar) left irc: Quit: Leaving.
[19:03] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) left irc: Quit: qrush
[19:05] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) joined #redis.
[19:08] sudoer (~jtoy@c-71-202-44-24.hsd1.ca.comcast.net) joined #redis.
[19:15] gnrfan (~gnrfan@190.234.136.133) left irc: Quit: Leaving
[19:17] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) left irc: Read error: Operation timed out
[19:18] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) joined #redis.
[19:22] bmizerany (~bmizerany@204.14.152.118) left irc: Remote host closed the connection
[19:29] nirvdrum (~nirvdrum@pool-173-48-56-214.bstnma.fios.verizon.net) left irc: Remote host closed the connection
[19:30] tilgovi (~quassel@2001:550:c00:8:222:19ff:fe0b:8e39) left irc: Read error: Connection reset by peer
[19:33] ttpva (~ttpva@a79-168-98-182.cpe.netcabo.pt) left irc: Quit: ttpva
[19:39] sudoer (~jtoy@c-71-202-44-24.hsd1.ca.comcast.net) left irc: Quit: sudoer
[19:45] geekondemand (~jeff@24-196-64-199.static.mdsn.wi.charter.com) left irc: Read error: Connection reset by peer
[19:46] mattly (~mattly@c-76-115-3-141.hsd1.or.comcast.net) joined #redis.
[19:53] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) left irc: Quit: senderista
[20:06] brandonleach (~brandonle@216.sub-75-208-208.myvzw.com) left irc: Ping timeout: 255 seconds
[20:07] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) joined #redis.
[20:13] carmony_ (~justin@c-24-10-194-172.hsd1.ut.comcast.net) joined #redis.
[20:13] carmony (~justin@c-24-10-194-172.hsd1.ut.comcast.net) left irc: Read error: Connection reset by peer
[20:13] carmony_ -> carmony
[20:20] curtisc (~curtisc@75-144-19-1-ca.sfba.hfc.comcastbusiness.net) left irc: Quit: curtisc
[20:22] hikeonpast (~hikeonpas@71-95-209-242.static.mtpk.ca.charter.com) left irc: Ping timeout: 276 seconds
[20:34] jdmaturen (~jdmaturen@204.16.157.170) left irc: Quit: jdmaturen
[20:37] qrush (~qrush@c-24-60-248-134.hsd1.ma.comcast.net) left irc: Read error: Connection reset by peer
[20:45] KevBurnsJr (~kevburnsj@c-98-234-48-188.hsd1.ca.comcast.net) joined #redis.
[20:48] curtisc (~curtisc@adsl-75-37-36-186.dsl.pltn13.sbcglobal.net) joined #redis.
[20:50] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) joined #redis.
[20:52] traceback0 (~traceback@c-67-180-79-10.hsd1.ca.comcast.net) left irc: Quit: traceback0
[20:54] senderista (~senderist@c-71-197-184-225.hsd1.wa.comcast.net) left irc: Client Quit
[20:54] curtisc (~curtisc@adsl-75-37-36-186.dsl.pltn13.sbcglobal.net) left irc: Quit: curtisc
[20:57] jdmaturen (~jdmaturen@c-67-160-216-89.hsd1.ca.comcast.net) joined #redis.
[20:59] fedesilva (~fedesilva@173-5.dedicado.com.uy) left irc: Quit: zZZzzZZzz
[21:07] hobodave (~hobodave@pdpc/supporter/professional/hobodave) left irc: Remote host closed the connection
[21:09] robotarmy (~o_o@pdpc/supporter/professional/robotarmy) left irc: Remote host closed the connection
[21:33] ardsrk (~c42engine@122.167.75.252) joined #redis.
[21:44] curtisc (~curtisc@adsl-75-37-36-186.dsl.pltn13.sbcglobal.net) joined #redis.
[21:51] hobodave (~hobodave@pdpc/supporter/professional/hobodave) joined #redis.
[21:51] ardsrk (c42engine@122.167.75.252) left #redis.
[21:53] blueadept (~blueadept@cpe-24-160-96-254.tampabay.res.rr.com) left irc: Quit: Leaving
[22:07] brandonleach (~brandonle@99-27-200-28.lightspeed.irvnca.sbcglobal.net) joined #redis.
[22:22] curtisc (~curtisc@adsl-75-37-36-186.dsl.pltn13.sbcglobal.net) left irc: Ping timeout: 260 seconds
[22:25] robstorrs (~robstorrs@dsl081-065-023.sfo1.dsl.speakeasy.net) joined #redis.
[22:28] unbracketed (~brian@cpe-76-169-191-136.socal.res.rr.com) left irc: Quit: Leaving.
[22:37] jdunck (~jdunck@cpe-72-190-124-161.tx.res.rr.com) left irc: Ping timeout: 246 seconds
[22:38] sudoer (~jtoy@c-98-210-164-172.hsd1.ca.comcast.net) joined #redis.
[22:44] brandonleach (~brandonle@99-27-200-28.lightspeed.irvnca.sbcglobal.net) left irc: Quit: brandonleach
[22:51] curtisc (~curtisc@99-8-186-153.lightspeed.snfcca.sbcglobal.net) joined #redis.
[23:04] marcostoledo (~marcostol@95.61.110.123) joined #redis.
[23:06] synk (~synk@ugate.dwango.co.jp) left irc: Quit: Leaving...
[23:14] synk (~synk@ugate.dwango.co.jp) joined #redis.
[23:15] Wombert (~Wombert@dslb-188-099-140-107.pools.arcor-ip.net) left irc: Quit: Wombert
[23:16] TomsB (~TomsB@46.109.55.4) joined #redis.
[23:18] mattly (~mattly@c-76-115-3-141.hsd1.or.comcast.net) left irc: Remote host closed the connection
[23:37] asenchi (~asenchi@206.173.142.114) left irc: Ping timeout: 240 seconds
[23:38] asenchi (~asenchi@206.173.142.114) joined #redis.
[23:42] tilgovi (~quassel@c-98-210-193-142.hsd1.ca.comcast.net) joined #redis.
[23:43] kost_ (~kost@217.172.23.200) joined #redis.
[23:48] johnsanders (~johnsande@204.16.157.170) left irc: Quit: johnsanders
[23:51] xgmlovebee (74f612f8@gateway/web/freenode/ip.116.246.18.248) joined #redis.
[23:53] xgmlovebee (74f612f8@gateway/web/freenode/ip.116.246.18.248) left #redis.
[00:00] --- Thu Jan 13 2011