#redis on irc.freenode.net 2011-11-23 (txt)

[00:00] horte (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[00:03] pharkmillups (~markphill@c-98-234-235-225.hsd1.ca.comcast.net) left irc: Quit: pharkmillups
[00:11] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) joined #redis.
[00:11] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Read error: Connection reset by peer
[00:11] horte (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[00:13] NetRoY (~NetRoY@nat/yahoo/x-onvunuodsmqvnswn) joined #redis.
[00:15] old_sound (~old_sound@77-58-253-248.dclient.hispeed.ch) joined #redis.
[00:25] thehodge (~thehodge@2001:470:1f09:13a3:7039:7609:18a0:4457) joined #redis.
[00:25] binaryten (~lloyd@62.82.228.34.static.user.ono.com) joined #redis.
[00:26] <horte> Hi Guys, I'm using the Jedis client to set hash values like this where each field has a value that looks like this [231 6 124 321]. it is a numerical array with approx. 500 integers. It takes about 8 seconds to put 1000 fields/values to the Hash. Is this correct or am I doing smth really wrong?
[00:26] <horte> Basically it looks like this {1=[321 421 543 321 53], 2=[95843 423 543 54 1 31 654]} etc. for 1000 values using HMSET
[00:28] <horte> The size of a value is ~3502 bytes
[00:28] KevBurnsJr (~kevburnsj@c-76-126-10-63.hsd1.ca.comcast.net) left irc:
[00:30] thehodge (~thehodge@2001:470:1f09:13a3:7039:7609:18a0:4457) left irc: Remote host closed the connection
[00:31] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[00:35] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Ping timeout: 240 seconds
[00:42] rittyan (~rittyan@95-26-215-143.broadband.corbina.ru) left irc: Remote host closed the connection
[00:49] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[00:51] Keanu_ (~Keanu@60-250-192-34.HINET-IP.hinet.net) left irc: Ping timeout: 252 seconds
[00:52] <mattbillenstein> horte, so 1000 key/values into a single hash?
[00:52] Keanu_ (~Keanu@60-250-192-34.HINET-IP.hinet.net) joined #redis.
[00:59] <horte> mattbillenstein: 50k key/values
[00:59] <horte> Stupid idea?
[00:59] <mattbillenstein> what is the top key?
[00:59] <mattbillenstein> seems sorta wonky — will you ever read them all at the same time?
[01:00] <horte> topkey? the keys are like ids. 1-50k
[01:00] <mattbillenstein> 50k 3500 byte keys is a lot of data
[01:00] <mattbillenstein> oh, I thought you meant one hash
[01:00] <horte> 3500 byte is the value
[01:00] <horte> Well its one hash
[01:00] <horte> Oh wait.
[01:00] <horte> Rewind
[01:00] <horte> I have one hash. Containing 50k+ fields
[01:01] <mattbillenstein> yeah
[01:01] <mattbillenstein> that seems odd
[01:01] <mattbillenstein> especially with largish values
[01:01] <horte> each fields has a value of an array eg. [21 31 53] that is 3500 bytes large
[01:01] <mattbillenstein> yeah, so what is the top key then?
[01:02] <mattbillenstein> and do you ever get all the values at the same time?
[01:02] <horte> oh index:collection:1
[01:02] <horte> and index:collection:2
[01:02] <horte> etc
[01:02] <horte> Yeah I want to get all the values at once
[01:03] <horte> I want to be able to get all fields and all values into two arrays and map them accordingly
[01:03] <mattbillenstein> that'll be like 175 MB of data
[01:03] <horte> like hvals
[01:03] <mattbillenstein> 50k * 3500
[01:03] <horte> Yeah
[01:04] <horte> So my use case is this:
[01:05] <horte> I have these arrays that I need to fetch as fast as possible. The arrays are grouped into collections and a collection could have from 1 -> x Million arrays
[01:05] <horte> Each array has a unique id for a collection
[01:06] <horte> So I need to be able to map the field/key to the value
[01:06] <horte> Is a hash the wrong way to go?
[01:07] <mattbillenstein> is this like a join for something else?
[01:08] <mattbillenstein> well, hash lookup/set is O(N)
[01:08] <mattbillenstein> so yeah, I think that's gonna be slow
[01:08] <mattbillenstein> other thing is, you can probably save a lot of space by encoding the arrays somehow
[01:08] <mattbillenstein> are they ids?
[01:08] mlangenberg (~mlangenbe@77.222.68.18) left irc: Remote host closed the connection
[01:08] <horte> yeah ids
[01:08] <mattbillenstein> 64bit or something?
[01:09] <horte> and each id is mapped to another hashfield
[01:09] <horte> in another hash
[01:09] <mattbillenstein> are you mapping something relational here?
[01:09] <horte> just number from the range 1- ~5M
[01:09] <horte> ye
[01:09] <mattbillenstein> so 4 bytes would do?
[01:10] <mattbillenstein> I'd encode and possibly compress the values
[01:10] <horte> I guess
[01:10] <horte> Still use a hash or should I think of using regular keys?
[01:10] <mattbillenstein> hash is O(N)
[01:10] <mattbillenstein> so if you wanna put 1MM keys in a single hash — that's gonna be silly slow
[01:11] <mattbillenstein> err
[01:11] <mattbillenstein> wait
[01:11] <mattbillenstein> I'm wrong
[01:11] <mattbillenstein> its O(N) for the number of keys you are getting
[01:12] <mattbillenstein> so it's constant
[01:12] <mattbillenstein> you said 1000 keys took 8s?
[01:12] <horte> 5000 keys
[01:12] <horte> to set
[01:12] <horte> not to read
[01:14] <mattbillenstein> taht seems a bit slow — that's like 17.5MB of data — so going over the wire it should take like < 1s
[01:14] <horte> Yeah, and running all on my local.
[01:15] <horte> Need to re run that again (perhaps with another client). Anyways it should still be possible to use a hash for my case?
[01:15] <horte> I mean the right way to go
[01:16] <horte> Though If I run HVALS and HKEYS will they both return the response in the same order?
[01:16] <horte> Else i'm pretty f*cked
[01:17] <mattbillenstein> dunno if the hash guarantees order
[01:17] <mattbillenstein> do you ever read the values individually?
[01:17] <horte> Yeah, does that too
[01:18] <horte> Depending on if a jobs needs just one specific array or all of them.
[01:18] wam (~wam@unaffiliated/wam) joined #redis.
[01:18] <mattbillenstein> http://paste.pocoo.org/show/5111796/
[01:19] <mattbillenstein> how many values are in that 3500 byte array?
[01:19] <horte> "Page Not Found"
[01:19] <horte> 500 integers
[01:19] <horte> with a space between
[01:19] <horte> could be more or less, but mean is 500
[01:20] <mattbillenstein> yeah, so if you can pack it, it becomes like 2k then
[01:20] <mattbillenstein> which is a little more than half
[01:21] <mattbillenstein> that's an easy optimization
[01:21] <horte> alright
[01:21] <mattbillenstein> how often do you read?
[01:21] <horte> All the time
[01:21] <mattbillenstein> if you compress it, it'll probably be another 50% smaller
[01:21] <mattbillenstein> like in a web request?
[01:22] <mattbillenstein> like 50k keys will be like 40MB encoded/compressed — that's a lot of data to be pulling — the I/O will kill you
[01:22] <mattbillenstein> unless you cache on the application or something
[01:22] <horte> When an API request comes in it reads
[01:22] <mattbillenstein> how much data do you return per request?
[01:23] <mattbillenstein> like you read 100MB of data and return 100KB?
[01:23] <mattbillenstein> probably storing it wrong if that's the case
[01:23] <horte> To redis? Nothing. To client just couple of kb
[01:23] <mattbillenstein> yeah, that's insane
[01:23] <horte> Textanalysis ... :)
[01:23] <mattbillenstein> what is the worse case?
[01:24] <mattbillenstein> like 1MM keys in a hash?
[01:24] <mattbillenstein> each value is like 2k?
[01:24] <horte> Probably don't need to get all keys. Just a subset
[01:25] <horte> Out of 2 Million values I know that I need to get about 100k.
[01:25] <mattbillenstein> you need the database to filter that down for you
[01:26] <mattbillenstein> pulling the whole hash and doing it in the app is gonna be super expensive per request
[01:26] <horte> The filtering is handled by some clustering algoritms in my application
[01:26] <mattbillenstein> like it'll take several (tens) of seconds per request no?
[01:27] <horte> We are using MySQL today and it takes < 2 s to do all that
[01:27] <horte> But ofc we have some in the application memory
[01:28] <mattbillenstein> are you like sorting/paging or something?
[01:28] <mattbillenstein> are you moving that logic out of mysql and into the app when using redis?
[01:29] <horte> Lets say I have 1M documents and then i cluster them into X number of clusters.
[01:29] <horte> That is done by the application
[01:29] <horte> and all clusters is stored with relations
[01:30] <horte> So I know what cluster to get and know I need all documents in that cluster
[01:30] <horte> that is what I want redis for.
[01:30] <horte> Fetching all documents (numerical arrays) from redis
[01:30] <horte> Instead of fetching them from MySQL
[01:31] <mattbillenstein> ic
[01:31] <mattbillenstein> how do you build all the clusters and so forth?
[01:33] lstrojny (~lstrojny@p5099f5c8.dip0.t-ipconnect.de) joined #redis.
[01:33] <horte> That is done algorithmically
[01:34] <horte> But to be able to that I also need to fetch the arrays from redis
[01:34] drbobbeaty (~drbobbeat@c-67-184-75-162.hsd1.il.comcast.net) joined #redis.
[01:34] <mattbillenstein> wonder if you can do the filtering in redis using sets or something
[01:34] <horte> So step 1) put all document arrays to redis 2) fetch them from redis but not all at once
[01:36] <horte> So when thinking about it we will not fetch 1M in one call. But a couple of thousands
[01:36] <mattbillenstein> 200k 3.5k fields?
[01:36] <horte> yeah
[01:37] <horte> :S
[01:37] <mattbillenstein> that's 700MB
[01:37] <mattbillenstein> ??
[01:37] <mattbillenstein> just moving 700MB across the wire will take like a minute
[01:37] <mattbillenstein> how do you do this with mysql in 2s?
[01:38] pouete (~julien@unaffiliated/pouete) left irc: Remote host closed the connection
[01:39] <horte> As it is today we have another step which we want to remove and do on the fly
[01:42] <horte> I mean selecting 5000 rows from mysql takes < 100ms
[01:42] <mattbillenstein> what's in a row?
[01:43] <horte> a bunch of stuff. including numerical arrays, dates, ids, etc
[01:43] <mattbillenstein> like how much data?
[01:43] <mattbillenstein> 100 bytes?
[01:44] old_sound (~old_sound@77-58-253-248.dclient.hispeed.ch) left irc: Read error: Connection reset by peer
[01:44] old_sound (~old_sound@77-58-253-248.dclient.hispeed.ch) joined #redis.
[01:44] <horte> approx 2kb
[01:45] <mattbillenstein> and you pull all the fields?
[01:45] <mattbillenstein> 5000 * 2k is 10MB
[01:46] <mattbillenstein> 200k * 3.5k is 667MB
[01:46] <mattbillenstein> sounds like you're paging in mysql?
[01:46] <mattbillenstein> and you're not paging in redis
[01:47] m0rpho (b0c66953@gateway/web/freenode/ip.176.198.105.83) left irc: Quit: Page closed
[01:47] hackband (~hackband@0x5359d382.cpe.ge-1-1-0-1104.bynqu1.customer.tele.dk) joined #redis.
[01:47] <horte> Ah
[01:48] <mattbillenstein> like you've changed how you map the problem
[01:48] <mattbillenstein> so you're fighting I/O
[01:48] <mattbillenstein> between the db and the app
[01:48] <mattbillenstein> or something, I dunno
[01:50] <mattbillenstein> so if you need to page these lists of ids
[01:50] <mattbillenstein> you probably want to do it in redis using lists or something
[01:54] <horte> Ok. Thanks!
[01:54] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) joined #redis.
[01:54] <horte> Back to the drawing board
[01:54] <mattbillenstein> look at the redis data structures
[01:54] <mattbillenstein> they're generally richer than what you can do with tables on mysql
[01:55] <mattbillenstein> so you should be able to compose your data more efficiently in redis I think
[01:55] <mattbillenstein> well, but you don't get joins in redis really
[01:55] <mattbillenstein> so not sure what the key thing for your problem is
[01:56] <horte> yeah i need to think thru this
[01:57] <mattbillenstein> word
[01:58] ron (~ron@unaffiliated/ron) left irc: Quit: ZNC - http://znc.in
[01:58] ron (~ron@unaffiliated/ron) joined #redis.
[02:04] melle (589f0b9b@gateway/web/freenode/ip.88.159.11.155) joined #redis.
[02:14] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) left irc: Read error: No route to host
[02:14] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) joined #redis.
[02:19] wam (~wam@unaffiliated/wam) left irc: Ping timeout: 258 seconds
[02:25] drbobbeaty (~drbobbeat@c-67-184-75-162.hsd1.il.comcast.net) left irc: Quit: drbobbeaty
[02:29] Net_RoY (~NetRoY@nat/yahoo/x-wzdmgtztdhnovgwk) joined #redis.
[02:30] d0k (~d0k@aef.wh.uni-dortmund.de) joined #redis.
[02:31] NetRoY (~NetRoY@nat/yahoo/x-onvunuodsmqvnswn) left irc: Ping timeout: 240 seconds
[02:31] Net_RoY -> NetRoY
[02:32] lupisak (~lupisak@93-97-52-241.zone5.bethere.co.uk) left irc: Ping timeout: 276 seconds
[02:33] __alex (~alex@business-178-015-083-042.static.arcor-ip.net) joined #redis.
[02:37] Keanu_ (~Keanu@60-250-192-34.HINET-IP.hinet.net) left irc: Ping timeout: 244 seconds
[02:37] Kosma (kosma@host-6-66.internetunion.pl) joined #redis.
[02:41] tim_smart|away (~tim@fostle.xen.prgmr.com) left irc: Ping timeout: 276 seconds
[02:42] tim_smart|away (~tim@fostle.xen.prgmr.com) joined #redis.
[02:44] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) left irc: Quit: Computer has gone to sleep.
[02:56] stephank (~stephank@82.197.214.45) joined #redis.
[03:25] FunkyBob (~curtis@li45-68.members.linode.com) left #redis.
[03:27] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[03:28] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[03:32] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Ping timeout: 248 seconds
[03:33] Wombert (~Wombert@munich.bitextender.net) joined #redis.
[03:35] seivan (~seivan@bb115-66-151-111.singnet.com.sg) left irc: Remote host closed the connection
[03:38] NetRoY (~NetRoY@nat/yahoo/x-wzdmgtztdhnovgwk) left irc: Quit: NetRoY
[03:46] drbobbeaty (~drbobbeat@38.98.137.29) joined #redis.
[03:47] marcostoledo (~marcostol@189.29.90.157) joined #redis.
[03:58] ron (~ron@unaffiliated/ron) left irc: Quit: ZNC - http://znc.in
[03:59] ron (~ron@unaffiliated/ron) joined #redis.
[04:04] horte_ (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[04:05] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Read error: Operation timed out
[04:05] horte_ -> horte
[04:05] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Remote host closed the connection
[04:05] horte (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[04:06] tim_smart|away (~tim@fostle.xen.prgmr.com) left irc: Ping timeout: 252 seconds
[04:07] tim_smart|away (~tim@fostle.xen.prgmr.com) joined #redis.
[04:09] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) joined #redis.
[04:09] elcuervo (~elcuervo@r186-48-213-182.dialup.adsl.anteldata.net.uy) joined #redis.
[04:11] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) left irc: Remote host closed the connection
[04:12] coco_m (~mark@89.100.113.252) joined #redis.
[04:14] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Quit: horte
[04:15] oskarols_ (~Olsson@static-212-162-171-110.cust.tele2.se) joined #redis.
[04:15] oskarols (~Olsson@static-212-162-171-110.cust.tele2.se) left irc: Read error: Connection reset by peer
[04:15] oskarols_ -> oskarols
[04:15] thehodge_ (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) joined #redis.
[04:16] thehodge_ (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) left irc: Remote host closed the connection
[04:18] saroj (6eea75c2@gateway/web/freenode/ip.110.234.117.194) left irc: Quit: Page closed
[04:24] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) joined #redis.
[04:34] Nuck -> NuckOff
[04:35] horte (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[04:35] petercooper (~petercoop@82.144.254.2) joined #redis.
[04:37] abecc (~abecc@190.18.132.2) joined #redis.
[04:38] NuckOff -> Nuck
[04:41] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) left irc: Remote host closed the connection
[04:41] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) joined #redis.
[04:45] steph021 (~steph021@unaffiliated/steph021) left irc: Quit: Computer has gone to sleep.
[04:45] thehodge (~thehodge@host109-157-255-134.range109-157.btcentralplus.com) left irc: Ping timeout: 240 seconds
[04:54] Nuck -> NuckOff
[04:59] seppo0010 (~Adium@200.69.194.105) joined #redis.
[05:06] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[05:08] thehodge (~thehodge@82.109.33.196) joined #redis.
[05:14] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) left irc: Read error: No route to host
[05:15] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) joined #redis.
[05:22] thehodge (~thehodge@82.109.33.196) left irc: Ping timeout: 258 seconds
[05:22] lstrojny_ (~lstrojny@p5099f5c8.dip0.t-ipconnect.de) joined #redis.
[05:24] lstrojny (~lstrojny@p5099f5c8.dip0.t-ipconnect.de) left irc: Ping timeout: 245 seconds
[05:24] lstrojny_ -> lstrojny
[05:25] thehodge (~thehodge@ip-87-82-192-132.easynet.co.uk) joined #redis.
[05:27] thehodge_ (~thehodge@82.109.33.196) joined #redis.
[05:27] thehodge_ (~thehodge@82.109.33.196) left irc: Read error: Connection reset by peer
[05:27] thehodge_ (~thehodge@82.109.33.196) joined #redis.
[05:30] thehodge (~thehodge@ip-87-82-192-132.easynet.co.uk) left irc: Ping timeout: 252 seconds
[05:33] jano (~djanowski@186.136.173.14) joined #redis.
[05:33] fmeyer (~fmeyer@187-27-240-142.3g.claro.net.br) joined #redis.
[05:34] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Quit: horte
[05:35] seppo0010 (~Adium@200.69.194.105) left #redis.
[05:40] horte (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[05:48] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 252 seconds
[05:53] elcuervo_ (~elcuervo@r186-48-212-105.dialup.adsl.anteldata.net.uy) joined #redis.
[05:54] Wombert (~Wombert@munich.bitextender.net) left irc: Read error: Connection reset by peer
[05:54] Wombert (~Wombert@munich.bitextender.net) joined #redis.
[05:54] PiotrSikora (~none@nginx/adept/piotrsikora) left irc: Ping timeout: 240 seconds
[05:54] fmeyer (~fmeyer@187-27-240-142.3g.claro.net.br) left irc: Ping timeout: 244 seconds
[05:54] xetorthio (~jleibiusk@200.41.238.50) joined #redis.
[05:56] PiotrSikora (~none@nginx/adept/piotrsikora) joined #redis.
[05:57] elcuervo (~elcuervo@r186-48-213-182.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 244 seconds
[05:58] jano (~djanowski@186.136.173.14) left irc: Disconnected by services
[05:58] jano_ (~djanowski@host186.190-136-33.telecom.net.ar) joined #redis.
[05:59] soveran (~soveran@186.19.214.247) joined #redis.
[06:00] moshef (~moshef@84.94.109.182.cable.012.net.il) joined #redis.
[06:00] soveran (~soveran@186.19.214.247) left irc: Read error: Connection reset by peer
[06:01] fmeyer (~fmeyer@201.87.5.103.user.ajato.com.br) joined #redis.
[06:04] Comandante (~Comandant@200-42-23-2.dup.prima.net.ar) joined #redis.
[06:05] thehodge_ (~thehodge@82.109.33.196) left irc: Remote host closed the connection
[06:06] thehodge (~thehodge@82.109.33.196) joined #redis.
[06:10] thehodge (~thehodge@82.109.33.196) left irc: Ping timeout: 258 seconds
[06:13] bubu\z (~bubu\a@ec2-46-137-119-156.eu-west-1.compute.amazonaws.com) left irc: Remote host closed the connection
[06:13] donspaulding (~donspauld@174-156-52-242.pools.spcsdns.net) joined #redis.
[06:22] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[06:22] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[06:22] steph021 (~steph021@unaffiliated/steph021) joined #redis.
[06:24] steph021 (~steph021@unaffiliated/steph021) left irc: Client Quit
[06:25] steph021 (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[06:25] steph021 (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[06:25] steph021 (~capt_crun@unaffiliated/steph021) joined #redis.
[06:26] donspaulding (~donspauld@174-156-52-242.pools.spcsdns.net) left irc: Quit: Leaving.
[06:26] steph021 (~capt_crun@unaffiliated/steph021) left irc: Client Quit
[06:27] awksed2greep (~awksedgre@c-24-98-243-58.hsd1.ga.comcast.net) left irc: Quit: This computer has gone to sleep
[06:29] steph021 (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[06:29] steph021 (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[06:29] steph021 (~capt_crun@unaffiliated/steph021) joined #redis.
[06:29] steph021 (~capt_crun@unaffiliated/steph021) left irc: Client Quit
[06:29] capt_crunch___ (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[06:29] capt_crunch___ (~capt_crun@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[06:29] capt_crunch___ (~capt_crun@unaffiliated/steph021) joined #redis.
[06:31] capt_crunch___ (~capt_crun@unaffiliated/steph021) left irc: Client Quit
[06:32] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[06:32] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[06:32] steph021 (~steph021@unaffiliated/steph021) joined #redis.
[06:37] jscheel (~jscheel@corp.centresource.com) joined #redis.
[06:37] jscheel (~jscheel@corp.centresource.com) left irc: Changing host
[06:37] jscheel (~jscheel@drupal.org/user/116197/view) joined #redis.
[06:39] jrz (~jrz@87.239.184.141) joined #redis.
[06:39] melle (589f0b9b@gateway/web/freenode/ip.88.159.11.155) left irc: Quit: Page closed
[06:42] ceej_ (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[06:43] jano_ -> jano
[06:44] fmeyer (~fmeyer@201.87.5.103.user.ajato.com.br) left irc: Ping timeout: 276 seconds
[06:47] ceej_ (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Client Quit
[06:47] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[06:53] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Quit: ceej
[06:53] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[06:54] advt (~mehaves@you-just.gotpwned.com) left irc: Read error: Connection reset by peer
[06:54] advt_ (~mehaves@you-just.gotpwned.com) joined #redis.
[07:00] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Quit: ceej
[07:02] Geffy (~geoffgars@2a02:40:2:0:b9f3:2b86:4e5c:495c) left irc: Quit: leaving
[07:06] steph021 (~steph021@unaffiliated/steph021) left irc: Quit: Computer has gone to sleep.
[07:17] gtuhl (~gtuhl@12.166.56.146) joined #redis.
[07:27] soveran (~soveran@186.19.214.247) joined #redis.
[07:27] powdahound (~garret@beautylish.static.monkeybrains.net) joined #redis.
[07:28] skiold (~adrian@213.27.225.221) joined #redis.
[07:31] jano (~djanowski@host186.190-136-33.telecom.net.ar) left irc: Remote host closed the connection
[07:33] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[07:36] elcuervo (~elcuervo@r186-49-3-37.dialup.adsl.anteldata.net.uy) joined #redis.
[07:38] rsampaio (~rodrigo@187.45.255.3) joined #redis.
[07:39] elcuervo_ (~elcuervo@r186-48-212-105.dialup.adsl.anteldata.net.uy) left irc: Ping timeout: 260 seconds
[07:46] seppo00101 (~Adium@190.195.17.56) joined #redis.
[07:50] seppo00101 (~Adium@190.195.17.56) left #redis.
[07:51] steph021 (~steph021@unaffiliated/steph021) joined #redis.
[07:57] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) joined #redis.
[08:01] oskarols (~Olsson@static-212-162-171-110.cust.tele2.se) left irc: Quit: oskarols
[08:03] Sarevok (~locke@rrcs-98-101-176-143.midsouth.biz.rr.com) joined #redis.
[08:05] rittyan (~rittyan@2a02:6b8:0:40c:7aca:39ff:feb3:d779) left irc: Remote host closed the connection
[08:06] donspaulding (~donspauld@c-67-173-34-83.hsd1.il.comcast.net) joined #redis.
[08:07] stephank (~stephank@82.197.214.45) left irc: Ping timeout: 244 seconds
[08:08] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) left irc: Quit: jtsnow
[08:09] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) joined #redis.
[08:11] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) left irc: Client Quit
[08:11] seppo0010 (~Adium@200.69.194.105) joined #redis.
[08:13] old_sound (~old_sound@77-58-253-248.dclient.hispeed.ch) left irc: Quit: old_sound
[08:13] k3VB6u4d (~k3VB6u4d@74-95-35-197-Oregon.hfc.comcastbusiness.net) joined #redis.
[08:14] seppo0010 (~Adium@200.69.194.105) left irc: Read error: Connection reset by peer
[08:14] seppo0010 (~Adium@200.69.194.105) joined #redis.
[08:15] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) joined #redis.
[08:18] k3VB6u4d (~k3VB6u4d@74-95-35-197-Oregon.hfc.comcastbusiness.net) left irc: Ping timeout: 248 seconds
[08:19] k3VB6u4d (~k3VB6u4d@ec2-204-236-142-90.us-west-1.compute.amazonaws.com) joined #redis.
[08:20] soveran_ (~soveran@186.19.214.247) joined #redis.
[08:20] soveran (~soveran@186.19.214.247) left irc: Read error: Connection reset by peer
[08:21] grampajoe (~grampa@cpe-184-57-88-36.columbus.res.rr.com) joined #redis.
[08:23] p2w (~textual@rrcs-67-79-200-115.sw.biz.rr.com) joined #redis.
[08:23] jrz (~jrz@87.239.184.141) left irc: Quit: http://www.damianfiligree.com
[08:26] tjholowaychuk (~tjholoway@S010658b03565e6c6.gv.shawcable.net) joined #redis.
[08:26] tjholowaychuk (~tjholoway@S010658b03565e6c6.gv.shawcable.net) left irc: Excess Flood
[08:26] tjholowaychuk (~tjholoway@S010658b03565e6c6.gv.shawcable.net) joined #redis.
[08:29] seppo00101 (~Adium@190.195.17.56) joined #redis.
[08:30] seppo0010 (~Adium@200.69.194.105) left irc: Read error: Connection reset by peer
[08:31] perezd (~derek@99-105-56-161.lightspeed.sntcca.sbcglobal.net) left irc: Quit: perezd
[08:38] p1d (~p1d@ns1.vonaffenfels.de) left irc: Quit: ...
[08:50] jrz (~jrz@d152222.upc-d.chello.nl) joined #redis.
[08:52] seppo00101 (~Adium@190.195.17.56) left irc: Read error: Connection reset by peer
[08:53] seppo0010 (~Adium@190.195.17.56) joined #redis.
[08:54] horte_ (~horte@static-212-162-171-110.cust.tele2.se) joined #redis.
[08:54] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) left irc: Quit: jtsnow
[08:55] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) joined #redis.
[08:56] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Read error: Operation timed out
[08:56] horte_ -> horte
[09:00] seppo0010 (~Adium@190.195.17.56) left irc: Read error: Connection reset by peer
[09:01] seppo0010 (~Adium@190.195.17.56) joined #redis.
[09:03] rittyan (~rittyan@95-26-215-143.broadband.corbina.ru) joined #redis.
[09:04] Xinil (Xinil@cpe-76-90-66-61.socal.res.rr.com) joined #redis.
[09:05] <Xinil> I have two PHP clients subscribed to one particular channel. When I push something to the channel, both clients perform the action, instead of just one :(
[09:05] <Xinil> this is not supposed to happen is it?
[09:13] _macro (~Neil@c-67-169-183-90.hsd1.ca.comcast.net) joined #redis.
[09:15] vkandy (~vkandy@static-68-179-6-233.ptr.terago.net) joined #redis.
[09:16] gregp76 (~gregpendl@c-24-7-25-59.hsd1.ca.comcast.net) joined #redis.
[09:17] <ron> why not?
[09:17] seppo0010 (~Adium@190.195.17.56) left #redis.
[09:18] <Xinil> i must have completely missed the boat on the pub/sub > queuing documentation
[09:18] <Xinil> does the pub/sub not work in such a way that I can have multiple clients and each will run the message if it's not busy?
[09:18] <Xinil> i.e., queue the messages until a client is available?
[09:22] <Xinil> hm, after further reading, i see redis doesn't natively support competing consumers
[09:24] soveran_ (~soveran@186.19.214.247) left irc: Remote host closed the connection
[09:24] skiold (~adrian@213.27.225.221) left irc: Quit: Get MacIrssi - http://www.sysctl.co.uk/projects/macirssi/
[09:26] seppo00101 (~Adium@200.69.194.105) joined #redis.
[09:27] seppo00101 (~Adium@200.69.194.105) left #redis.
[09:27] <ron> normally, pub/sub means that anyone can subscribe, and everyone that subscribes gets a copy.
[09:27] <ron> it's not a queue.
[09:28] binaryten (~lloyd@62.82.228.34.static.user.ono.com) left irc: Ping timeout: 260 seconds
[09:28] seppo0010 (~Adium@190.195.17.56) joined #redis.
[09:30] seppo0010 (~Adium@190.195.17.56) left #redis.
[09:38] KevBurnsJr (~KevBurnsJ@50.0.103.39) joined #redis.
[09:47] grampajoe (~grampa@cpe-184-57-88-36.columbus.res.rr.com) left irc: Quit: Bye!
[09:49] brl0 (~shiftins@173-8-169-73-SFBA.hfc.comcastbusiness.net) joined #redis.
[09:51] seppo00101 (~Adium@200.69.194.105) joined #redis.
[09:54] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Quit: ceej
[09:57] agile (~mike@71-86-124-96.dhcp.ftwo.tx.charter.com) joined #redis.
[10:00] moshef (~moshef@84.94.109.182.cable.012.net.il) left irc: Quit: moshef
[10:04] seppo00101 (~Adium@200.69.194.105) left #redis.
[10:10] petercooper (~petercoop@82.144.254.2) left irc: Remote host closed the connection
[10:12] coco_m (~mark@89.100.113.252) left irc: Quit: coco_m
[10:15] hackband (~hackband@0x5359d382.cpe.ge-1-1-0-1104.bynqu1.customer.tele.dk) left irc: Remote host closed the connection
[10:24] vkandy (~vkandy@static-68-179-6-233.ptr.terago.net) left irc: Ping timeout: 244 seconds
[10:24] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) joined #redis.
[10:24] <gtuhl> the queue operations in redis would work great for that, I have a queue that handles millions of messages a day with ~100 consumers doing BRPOPs and no issues
[10:28] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[10:35] pharkmillups (~markphill@70-36-146-239.dsl.dynamic.sonic.net) joined #redis.
[10:36] hahuang65 (~hahuang65@204.11.231.186.static.etheric.net) joined #redis.
[10:38] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[10:38] coco_m (~mark@089-101-050037.ntlworld.ie) joined #redis.
[10:39] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[10:41] thehodge_ (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[10:41] coco_m (~mark@089-101-050037.ntlworld.ie) left irc: Client Quit
[10:41] gnrfan (~gnrfan@201.230.207.32) joined #redis.
[10:43] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Ping timeout: 248 seconds
[10:46] thehodge_ (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[10:47] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[10:47] gregp76 (~gregpendl@c-24-7-25-59.hsd1.ca.comcast.net) left #redis.
[10:47] horte (~horte@static-212-162-171-110.cust.tele2.se) left irc: Quit: horte
[10:50] Wombert (~Wombert@munich.bitextender.net) left irc: Quit: Wombert
[10:52] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[10:52] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[10:59] <ron> gtuhl: right, only the queue works by pull and I imagine Xinil is looking for push.
[11:03] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) left irc: Quit: martincozzi
[11:04] perezd (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) joined #redis.
[11:05] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) joined #redis.
[11:06] kushal (kdas@fedora/kushal) left irc: Ping timeout: 244 seconds
[11:06] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) left irc: Client Quit
[11:06] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[11:07] brl0 (~shiftins@173-8-169-73-SFBA.hfc.comcastbusiness.net) left irc: Quit: Lost terminal
[11:07] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) joined #redis.
[11:07] hahuang65 (~hahuang65@204.11.231.186.static.etheric.net) left irc: Quit: Textual IRC Client: http://www.textualapp.com/
[11:10] vkandy (~vkandy@static-68-179-6-233.ptr.terago.net) joined #redis.
[11:11] thehodge (~thehodge@cpc1-seac12-0-0-cust290.7-2.cable.virginmedia.com) left irc: Ping timeout: 248 seconds
[11:15] p1d (~p1d@e180016114.adsl.alicedsl.de) joined #redis.
[11:17] fmeyer (~fmeyer@201.87.5.103.user.ajato.com.br) joined #redis.
[11:19] senderista (~senderist@216.161.248.54) joined #redis.
[11:20] p2w (~textual@rrcs-67-79-200-115.sw.biz.rr.com) left irc: Quit: Leaving...
[11:33] jrz (~jrz@d152222.upc-d.chello.nl) left irc: Quit: http://www.damianfiligree.com
[11:35] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) joined #redis.
[11:37] airhorns (~airhorns@206-248-184-147.dsl.teksavvy.com) joined #redis.
[11:38] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[11:38] powdahound (~garret@beautylish.static.monkeybrains.net) left irc: Read error: Connection reset by peer
[11:39] contempt_ (~Hill@host39-157-dynamic.51-79-r.retail.telecomitalia.it) joined #redis.
[11:39] powdahound (~garret@beautylish.static.monkeybrains.net) joined #redis.
[11:44] Wombert (~Wombert@dslb-088-065-199-130.pools.arcor-ip.net) joined #redis.
[11:46] martincozzi (~martincoz@209.66.114.6) joined #redis.
[11:50] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[11:51] soveran (~soveran@186.19.214.247) joined #redis.
[11:58] felixge (~felixgeis@miranda/donor/theundefined) joined #redis.
[11:59] evilhackerdude -> ehd
[12:00] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[12:01] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[12:04] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) left irc: Quit: horte
[12:04] ehd -> evilhackerdude
[12:05] thehodge_ (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[12:05] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Ping timeout: 248 seconds
[12:11] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) joined #redis.
[12:14] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 244 seconds
[12:19] chrisjones (~chrisjone@ip68-224-245-47.lv.lv.cox.net) joined #redis.
[12:20] senderista (~senderist@216.161.248.54) left irc: Quit: senderista
[12:20] <chrisjones> Is there any way to publish a message to "all" channels?
[12:20] kennethlove (~kennethlo@kennethlove.xen.prgmr.com) joined #redis.
[12:21] hackband (~hackband@1385159443.dhcp.dbnet.dk) joined #redis.
[12:21] lstrojny (~lstrojny@p5099f5c8.dip0.t-ipconnect.de) left irc: Quit: lstrojny
[12:22] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) left irc: Quit: horte
[12:22] powdahound (~garret@beautylish.static.monkeybrains.net) left irc: Ping timeout: 244 seconds
[12:27] __alex (~alex@business-178-015-083-042.static.arcor-ip.net) left irc: Ping timeout: 240 seconds
[12:28] powdahound (~garret@beautylish.static.monkeybrains.net) joined #redis.
[12:30] thehodge_ (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[12:30] fmeyer_ (~fmeyer@201.87.5.103.user.ajato.com.br) joined #redis.
[12:30] chrisjones (~chrisjone@ip68-224-245-47.lv.lv.cox.net) left irc: Quit: Textual IRC Client: http://www.textualapp.com/
[12:31] fmeyer_ (~fmeyer@201.87.5.103.user.ajato.com.br) left irc: Client Quit
[12:33] gheegh (~wflanagan@pool-173-66-40-120.washdc.fios.verizon.net) joined #redis.
[12:34] <gheegh> hey all.. question.. i've got a redis-server that crashes every 2-3 days.. i checked the logs and its full of Can't save in background: fork: Cannot allocate memory what's the issue? GoogleFu is failing me
[12:34] <soveran> gheegh: it probably has to do with the amount of free memory your server has
[12:35] <gheegh> alright..
[12:35] <gheegh> as in ram, right?
[12:35] <soveran> it should be able to accommodate twice the size of your database
[12:35] <soveran> in ram, yes
[12:35] <Kosma> http://redis.io/topics/faq -> "Background saving is failing with a fork() error under Linux even if I've a lot of free RAM!"
[12:36] <Kosma> gheegh: how much RAM do you have? how much does Redis use? do you have a memory limit set? does your dmesg show any OOMs?
[12:37] <gheegh> Kosma: 8g of RAM.. we gave redis virtually all of it.. hold on on dmesg
[12:37] <gheegh> yes.. [238032.277593] Out of memory: kill process 614 (redis-server) score 75164 or a child
[12:38] <Kosma> yeah, you're pushing it too far
[12:38] <Kosma> basically, Redis will eat memory until it's killed
[12:38] <vsmatck> It's useful to understand how linux copy-on-write works when determining redis maxmemory.
[12:38] <gheegh> though if you check redis, it actually has a lot of memory free.. we're not filling it up i don't believe.
[12:38] <Kosma> gheegh: it could be a fragmentation issue
[12:39] <Kosma> once RSS grows, it can't shrink
[12:39] <gheegh> right
[12:39] <Kosma> then, some process allocates some memory and it hits the ceiling
[12:39] <Kosma> ...and OOM kills redis-server since it's the biggest one and probably got the highest OOM score
[12:39] <Kosma> I've found a good memorylimit for Redis to be essential
[12:40] <Kosma> a good start is total_memory - os_memory_usage - 1GB
[12:40] <Kosma> so that you have 1GB of headroom even when Redis uses all its limit
[12:42] <Kosma> gheegh: btw, what's your usage pattern?
[12:42] <Kosma> cache? data storage?
[12:43] <gheegh> pretty stock aws instance.. mounted disk where i'm saving backkups (400 gig). it's an M1.large instance.
[12:44] <Kosma> yeah, but what kind of data do you store in Redis?
[12:44] <Kosma> are you using expiry and/or eviction?
[12:45] <gheegh> Kosma: We set expirations yes.. we store mostly strings.. its in flight queueing for a web crawl, so we're storing pages as they move through the pipeline
[12:45] <gheegh> here's the log.. FYI http://pastie.org/2910919
[12:45] <gheegh> er.. not log, config.
[12:45] <gheegh> about 60% write/ 40% read
[12:46] <gheegh> good size data for the pages, then a LOT of counters type data
[12:46] <Kosma> well, the very first step should be setting the memorylimit
[12:46] <gheegh> so, mostly hashes, some sets, and lists
[12:46] <Kosma> and maxmemory-policy
[12:48] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) joined #redis.
[12:50] jscheel (~jscheel@drupal.org/user/116197/view) left irc: Quit: jscheel
[12:50] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) left irc: Client Quit
[12:52] <gheegh> ok.. I've done that.. 6G the default..
[12:53] p1d (~p1d@e180016114.adsl.alicedsl.de) left irc: Ping timeout: 240 seconds
[12:55] <Kosma> remember to set maxmemory-policy
[12:56] <Kosma> but do think it through - otherwire you might end up with inconsistent database due to eviction
[12:56] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[12:56] <Kosma> and by inconsistent I mean inconsistency in relations between keys - Redis itself doesn't get corrupted by that
[12:57] <Kosma> k, fallout time
[12:57] Kosma (kosma@host-6-66.internetunion.pl) left irc: Quit: reboot
[12:57] <gheegh> So, i've set the maxmemory to 6GB, the maxmemory-policy to volatile-lru, enabled VM with a Vm-Max-memory of 6GB
[12:58] <gheegh> i think that makes sense, given what I've read.. anyone?
[13:00] p1d (~p1d@e180022185.adsl.alicedsl.de) joined #redis.
[13:03] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) joined #redis.
[13:04] elcuervo (~elcuervo@r186-49-3-37.dialup.adsl.anteldata.net.uy) left irc: Remote host closed the connection
[13:08] binaryten (~lloyd@77.231.217.59) joined #redis.
[13:12] <vsmatck> So this is a cache? You can eject random stuff? Do you need persistence at all?
[13:15] <gheegh> vsmatck: I need a bit of persistence.. the crawls take 1-2 days for large sites.. if it burps and reboots, i'd like to not start over.. but beyond that, no
[13:15] <vsmatck> Hm. Does sound like you need it.
[13:16] <vsmatck> What you have to consider is the rate at which you write. After redis forks it will duplicate any memory that is written.
[13:16] <gheegh> right. so, here is my new config.. i think it's configured this way pretty much?
[13:16] <vsmatck> So if your write _rate_ will take you over your max memory before the forked process finishes then you'll have problems.
[13:16] <vsmatck> If it's a crawler it sounds like your write rate (bytes/second) is low?
[13:17] <gheegh> not to the queue..
[13:18] <gheegh> most of the time we pull a page then we push a page, and we keep stats all along the way. we have queues made of lists and sets
[13:18] <gheegh> http://www.pastie.org/2911043
[13:18] <vsmatck> Ah. True. That sounds like a high write rate.
[13:19] <gheegh> vsmatck: right now, the overall system is not working hard.. except when ti was trying to write to disk.. the cpu sits pretty consistently at under 10% utilization (via top, i've got 4cpus so 0.10)
[13:19] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) left irc: Remote host closed the connection
[13:19] horte (~horte@173.195.11.224) joined #redis.
[13:20] <vsmatck> Hm. Like someone said earlier a maxmemory of 1/2 physical memory is required if you modify 100% of the stuff in your redis server while the forked process is saving stuff to disk.
[13:20] jacobrask (~jacob@jacobrask.net) joined #redis.
[13:21] <vsmatck> The fact that you have max memory at 6gB implies that you will modify at most 1/3 of your data while the forked saving process is running.
[13:21] <vsmatck> Does that sound reasonable?
[13:22] <vsmatck> If you modify one thing multiple times it doesn't count. So a queue might not be super bad for writes.
[13:22] <gheegh> yeah, far less than that..
[13:22] <gheegh> we are running the system at about 1G right now
[13:22] <vsmatck> With copy-on-write you only get a copy when the thing is modified the first time. If you modify something 100 times it just has to store the original copy somewhere.
[13:22] <gheegh> we are nowhere near capacity.. it was just the writes to disk that killed us, I think.. when it waas trying to background save
[13:23] <vsmatck> Ah, yeah you said you had maxmemory at 8gB before with 8gB physical memory right?
[13:23] <gheegh> yes
[13:23] <gheegh> that's right
[13:23] <vsmatck> 6gB is starting to sound reasonably conservative to me.
[13:24] <gheegh> Ok.. Good. I will take conservative for a while.. :-) I can use some stability.. this redis crash cost us 2 days of crawls.. @ $1.20/hr... (AWS)
[13:24] <vsmatck> ouch, that's a lot of beer money.
[13:25] <gheegh> nah.. not much money.. but the 2 days lost is kinda sucky.
[13:25] <gheegh> but if we crash every 3 days, we'll never get anything done.
[13:31] hornairs (~airhorns@69-196-165-104.dsl.teksavvy.com) joined #redis.
[13:34] airhorns (~airhorns@206-248-184-147.dsl.teksavvy.com) left irc: Ping timeout: 248 seconds
[13:34] horte (~horte@173.195.11.224) left irc: Quit: horte
[13:35] jacobrask (~jacob@jacobrask.net) left #redis.
[13:38] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[13:40] d0k (~d0k@aef.wh.uni-dortmund.de) left irc: Quit: he does that
[13:42] lux___ (~lux@ppp-155-71.24-151.libero.it) joined #redis.
[13:44] p1d (~p1d@e180022185.adsl.alicedsl.de) left irc: Quit: ...
[13:46] seivan (~seivan@cm67.eta201.maxonline.com.sg) joined #redis.
[13:46] <seivan> Hey does anyone know if R.hgetall supports deserializing strings from the value?
[13:47] <seivan> Right now I get all, iterate through everything and manually parse json...
[13:47] Comandante (~Comandant@200-42-23-2.dup.prima.net.ar) left irc: Remote host closed the connection
[13:50] senderista (~senderist@216.161.248.54) joined #redis.
[13:58] devdazed (~russ_nyc@static-96-239-59-191.nycmny.fios.verizon.net) joined #redis.
[13:59] <gheegh> seivan: i'm doing that in ruby as well.. (manual parsing)
[14:01] <seivan> hmm
[14:01] <seivan> I decided to get rid of it and just go with pure SQL for that part
[14:02] <seivan> I will store id's in a SET
[14:02] <seivan> and fetch data from SQL
[14:02] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) joined #redis.
[14:02] <seivan> Doing a caching solution where I keep a server copy of what the client has on it's db
[14:02] <seivan> So when the client hits the API and downloads stuff. It later sends a ack request to the server telling the server to update the server-sided cache
[14:03] <seivan> ergo. The server can always tell the client what to add and what to remove
[14:03] <seivan> :) pretty neat
[14:03] <gheegh> seivan: yep
[14:03] <seivan> https://gist.github.com/b63a02472175c40b67bf
[14:05] horte (~horte@c-246be555.32-24-64736c16.cust.bredbandsbolaget.se) left irc: Remote host closed the connection
[14:05] horte (~horte@173.195.11.152) joined #redis.
[14:06] fmeyer (~fmeyer@201.87.5.103.user.ajato.com.br) left irc: Ping timeout: 258 seconds
[14:06] horte (~horte@173.195.11.152) left irc: Client Quit
[14:11] skrul (~skrul@75-101-56-124.dsl.static.sonic.net) joined #redis.
[14:12] perezd_ (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) joined #redis.
[14:14] <seivan> hmm
[14:14] <seivan> I need to do some intersects, and difference
[14:14] <seivan> Should I store in redis and do it there
[14:14] <seivan> Or in ruby?
[14:14] <rittyan> seivan: ruby sets prolly written in c anyway?
[14:15] <seivan> it's an array
[14:15] <seivan> and yeah probably
[14:15] <rittyan> sets aren't just arrays
[14:15] <rittyan> in fact they ought to be hashes
[14:15] <seivan> I know
[14:15] <seivan> I am just saying the one in ruby is an array
[14:15] <seivan> the data structure
[14:15] <seivan> I will be loading the set into memory as an array
[14:15] <rittyan> okay, you know it better than me (know no ruby)
[14:16] <seivan> At least that's what the redis client does anyway
[14:16] <seivan> https://gist.github.com/c56899da1928fd57a307
[14:16] perezd (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) left irc: Ping timeout: 245 seconds
[14:16] perezd_ -> perezd
[14:16] <rittyan> seivan: smembers in ruby client returns an array?
[14:16] <seivan> yes
[14:16] <rittyan> I am not happy about it
[14:17] <seivan> I am sure there is a very good reason for it
[14:17] <rittyan> probably there is :)
[14:17] <rittyan> interesting
[14:17] <seivan> I guess peoples familiarity
[14:18] <seivan> Anyway
[14:18] <rittyan> I suspected that sets are broken in ruby or there are five different libraries for sets
[14:18] <seivan> hahaha maybe
[14:18] <rittyan> No offense at all
[14:18] <seivan> Care, I do iOS mostly these days
[14:18] <seivan> Occasional backend in Ruby
[14:18] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[14:21] <seivan> Mostly bothered that the cleint does not have anything builtin to deserialize hash values
[14:21] <seivan> So instead of storing it in a single hash
[14:21] <seivan> I got two different sets
[14:21] <seivan> gah
[14:21] <rittyan> what do you mean?
[14:21] <seivan> I guess it's all for the good
[14:21] <rittyan> Are you sure you use the right redis client?
[14:22] <seivan> Yeah
[14:22] <rittyan> How does it work with hashes then? What's wrong?
[14:22] <seivan> I mean it is supposed to store an array as a string when its a hash value
[14:22] <seivan> But when I use hgetall I would like it to deserialize (as an option) whatever values inside the hash
[14:22] <rittyan> oh, I always pack values myself
[14:22] <seivan> yeah I guess you have to
[14:22] <rittyan> That way I know exactly what they look like
[14:22] <rittyan> what to expect, etc
[14:23] <seivan> Well I solved it by having two sets
[14:23] <seivan> instead of both sets in a hash
[14:30] johnsanders (~johnsande@c-69-181-21-76.hsd1.ca.comcast.net) left irc: Quit: Leaving...
[14:30] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) joined #redis.
[14:32] coco_m (~mark@089-101-050037.ntlworld.ie) joined #redis.
[14:34] marcostoledo (~marcostol@189.29.90.157) left irc: Quit: Computer has gone to sleep.
[14:34] contempt_ (~Hill@host39-157-dynamic.51-79-r.retail.telecomitalia.it) left #redis.
[14:37] marcostoledo (~marcostol@189.29.90.157) joined #redis.
[14:37] rsampaio (~rodrigo@187.45.255.3) left irc: Quit: Computer has gone to sleep.
[14:37] powdahound (~garret@beautylish.static.monkeybrains.net) left irc: Quit: Textual IRC Client: http://www.textualapp.com/
[14:39] wmoss|away -> wmoss
[14:43] <seivan> Hmm
[14:43] <seivan> What is the best way to copy a set over to another key?
[14:44] <seivan> sunionstore
[14:44] <rittyan> wow, never thought about it :-)
[14:45] <seivan> R.multi do
[14:45] <seivan> R.del key_friends_ids_cache
[14:45] <seivan> R.sunionstore key_friends_ids_cache, key_friend_ids
[14:45] <seivan> end
[14:51] marcostoledo (~marcostol@189.29.90.157) left irc: Quit: Computer has gone to sleep.
[15:03] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 244 seconds
[15:04] marcostoledo (~marcostol@189.29.90.157) joined #redis.
[15:07] johnsanders (~johnsande@209.66.114.6) joined #redis.
[15:10] johnsanders (~johnsande@209.66.114.6) left irc: Read error: Connection reset by peer
[15:10] rsampaio (~rodrigo@189.38.154.97) joined #redis.
[15:12] johnsanders (~johnsande@209.66.114.6) joined #redis.
[15:13] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) left irc: Quit: jtsnow
[15:14] drbobbeaty (~drbobbeat@38.98.137.29) left irc: Quit: drbobbeaty
[15:16] lux___ (~lux@ppp-155-71.24-151.libero.it) left irc: Quit: This computer has gone to sleep
[15:17] felixge (~felixgeis@miranda/donor/theundefined) left irc: Quit: felixge
[15:18] xetorthio (~jleibiusk@200.41.238.50) left irc: Quit: Ex-Chat
[15:25] devdazed (~russ_nyc@static-96-239-59-191.nycmny.fios.verizon.net) left irc: Quit: devdazed
[15:26] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[15:32] rittyan_ (~rittyan@95-26-215-143.broadband.corbina.ru) joined #redis.
[15:34] ambroff (~ambroff@209.66.114.6) joined #redis.
[15:35] rittyan (~rittyan@95-26-215-143.broadband.corbina.ru) left irc: Ping timeout: 260 seconds
[15:36] NuckOff (~Nuck@unaffiliated/nuck) left irc: Ping timeout: 276 seconds
[15:38] Nuck (~Nuck@unaffiliated/nuck) joined #redis.
[15:43] hackband (~hackband@1385159443.dhcp.dbnet.dk) left irc: Remote host closed the connection
[15:47] rsampaio (~rodrigo@189.38.154.97) left irc: Quit: Computer has gone to sleep.
[15:51] coco_m (~mark@089-101-050037.ntlworld.ie) left irc: Quit: coco_m
[15:52] coco_m (~mark@089-101-050037.ntlworld.ie) joined #redis.
[15:52] petercooper (~petercoop@82.144.254.2) joined #redis.
[15:53] coco_m (~mark@089-101-050037.ntlworld.ie) left irc: Client Quit
[15:55] k3VB6u4d (~k3VB6u4d@ec2-204-236-142-90.us-west-1.compute.amazonaws.com) left irc: Ping timeout: 260 seconds
[15:55] ngw (~ngw@174-31-141-230.tukw.qwest.net) joined #redis.
[16:00] ngw (~ngw@174-31-141-230.tukw.qwest.net) left irc: Client Quit
[16:01] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) joined #redis.
[16:03] coco_m (~mark@089-101-050037.ntlworld.ie) joined #redis.
[16:04] mjr_ (~mjr@cpe-66-91-117-138.hawaii.res.rr.com) joined #redis.
[16:09] pharkmillups (~markphill@70-36-146-239.dsl.dynamic.sonic.net) left irc: Quit: pharkmillups
[16:11] coco_m (~mark@089-101-050037.ntlworld.ie) left #redis.
[16:13] seivan (~seivan@cm67.eta201.maxonline.com.sg) left irc: Remote host closed the connection
[16:21] binaryten (~lloyd@77.231.217.59) left irc: Ping timeout: 276 seconds
[16:22] macabre (~macabre@cpe-72-177-80-242.austin.res.rr.com) joined #redis.
[16:22] wmoss -> wmoss|away
[16:23] Sarevok (~locke@rrcs-98-101-176-143.midsouth.biz.rr.com) left #redis.
[16:31] queonda (~queonda@user-0cdfrf1.cable.mindspring.com) joined #redis.
[16:37] perezd (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) left irc: Remote host closed the connection
[16:37] perezd (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) joined #redis.
[16:37] soveran (~soveran@186.19.214.247) left irc: Remote host closed the connection
[16:45] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 244 seconds
[16:48] steph021 (~steph021@unaffiliated/steph021) left irc: Quit: Computer has gone to sleep.
[17:26] rittyan_ (~rittyan@95-26-215-143.broadband.corbina.ru) left irc: Remote host closed the connection
[17:27] gnrfan (~gnrfan@201.230.207.32) left irc: Quit: This computer has gone to sleep
[17:29] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[17:30] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[17:31] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) joined #redis.
[17:31] steph021 (~steph021@adsl-68-125-50-109.dsl.pltn13.pacbell.net) left irc: Changing host
[17:31] steph021 (~steph021@unaffiliated/steph021) joined #redis.
[17:35] KevBurnsJr (~KevBurnsJ@50.0.103.39) left irc:
[17:40] jtsnow (~jtsnow@75-169-84-216.slkc.qwest.net) left irc: Quit: jtsnow
[17:43] mattbillenstein (~Adium@cpe-75-84-198-75.socal.res.rr.com) left irc: Quit: Leaving.
[17:43] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[17:55] queonda (~queonda@user-0cdfrf1.cable.mindspring.com) left irc: Quit: queonda
[17:56] marcostoledo (~marcostol@189.29.90.157) left irc: Remote host closed the connection
[17:56] marcostoledo (~marcostol@189.29.90.157) joined #redis.
[18:04] marcostoledo (~marcostol@189.29.90.157) left irc: Quit: Computer has gone to sleep.
[18:08] rsampaio (~rodrigo@189-69-150-102.dial-up.telesp.net.br) joined #redis.
[18:10] gnrfan (~gnrfan@190.232.0.137) joined #redis.
[18:11] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Quit: ceej
[18:17] arek_deepinit (~arek@195.189.91.22) joined #redis.
[18:29] perezd (~derek@108-71-92-105.lightspeed.sntcca.sbcglobal.net) left irc: Ping timeout: 245 seconds
[18:32] skrul (~skrul@75-101-56-124.dsl.static.sonic.net) left irc: Quit: skrul
[18:38] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 244 seconds
[18:39] martincozzi (~martincoz@209.66.114.6) left irc: Quit: martincozzi
[18:51] tjholowaychuk (~tjholoway@S010658b03565e6c6.gv.shawcable.net) left irc: Quit: Bye!
[19:36] senderista (~senderist@216.161.248.54) left irc: Quit: senderista
[19:50] martincozzi_ (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) joined #redis.
[19:52] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) joined #redis.
[19:55] grampajoe (~grampa@cpe-184-57-88-36.columbus.res.rr.com) joined #redis.
[19:55] martincozzi_ (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) left irc: Ping timeout: 252 seconds
[20:02] caaakeeey (~cakey@cpc6-enfi15-2-0-cust204.hari.cable.virginmedia.com) joined #redis.
[20:02] caaakeeey (~cakey@cpc6-enfi15-2-0-cust204.hari.cable.virginmedia.com) left #redis.
[20:03] Mensa180 (467febf9@gateway/web/freenode/ip.70.127.235.249) joined #redis.
[20:07] skrul (~skrul@76.14.75.186) joined #redis.
[20:10] petercooper (~petercoop@82.144.254.2) left irc: Remote host closed the connection
[20:20] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) joined #redis.
[20:21] ceej (~anonymous@cpe-72-190-40-203.tx.res.rr.com) left irc: Client Quit
[20:28] Nuck -> NuckOff
[20:33] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) joined #redis.
[20:35] johnsanders (~johnsande@209.66.114.6) left irc: Quit: Leaving...
[20:43] rsampaio (~rodrigo@189-69-150-102.dial-up.telesp.net.br) left irc: Quit: Computer has gone to sleep.
[20:48] gnrfan (~gnrfan@190.232.0.137) left irc: Quit: Leaving
[20:51] abecc (~abecc@190.18.132.2) left irc: Quit: abecc
[20:55] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) left irc: Quit: Computer has gone to sleep.
[21:03] NuckOff -> Nuck
[21:05] johnsanders (~johnsande@c-69-181-21-76.hsd1.ca.comcast.net) joined #redis.
[21:05] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) joined #redis.
[21:06] fmeyer (~fmeyer@186.220.4.64) joined #redis.
[21:23] fmeyer (~fmeyer@186.220.4.64) left irc: Remote host closed the connection
[21:25] felixge (~felixgeis@91-65-245-141-dynip.superkabel.de) joined #redis.
[21:25] felixge (~felixgeis@91-65-245-141-dynip.superkabel.de) left irc: Changing host
[21:25] felixge (~felixgeis@miranda/donor/theundefined) joined #redis.
[21:44] johnsanders (~johnsande@c-69-181-21-76.hsd1.ca.comcast.net) left irc: Quit: Leaving...
[21:45] dieselz29 (~dieselz29@c-67-190-101-48.hsd1.co.comcast.net) joined #redis.
[21:46] <dieselz29> is there a way to get all keys that are a sorted set?
[21:46] <ron> not unless you manage those in a list/set.
[21:47] <dieselz29> ah i see. good to know! thanks!
[21:47] <dieselz29> i'm brand new to redis, testing its performance for an analytics use case … its pretty remarkable how fast it is
[21:47] <ron> or you could prefix all your set keys with a specific word, and then use the KEYS command to get all of them. but that's a *really* bad idea.
[21:48] ron <3 redis
[21:49] kushal (kdas@fedora/kushal) joined #redis.
[21:49] hornairs (~airhorns@69-196-165-104.dsl.teksavvy.com) left irc: Read error: Operation timed out
[21:50] johnsanders (~johnsande@c-69-181-21-76.hsd1.ca.comcast.net) joined #redis.
[21:50] skrul (~skrul@76.14.75.186) left irc: Quit: skrul
[21:50] <dieselz29> my plan is to store page view data in a sorted set. key would be the property (subdomain in my case), score would be the day number, value would be the count - does this make sense?
[21:51] <ron> sure.
[21:51] <dieselz29> makes it fast & easy to increment everytime theres a page view, and easy to query for the last 30 days of pageview counts for example
[21:51] <ron> though it depends what you want to do with the data.
[21:51] <dieselz29> primarily graphs of the last 30 days
[21:52] <ron> well, you can use hashes for that more efficiently.
[21:52] <dieselz29> i figured since they are sorted, i would be able to query a time chunk easier
[21:52] <dieselz29> *faster
[21:52] <ron> nope, using hashes would be faster.
[21:53] <dieselz29> interesting. is there a resource you could point me to that explains why that is?
[21:53] <ron> Sure, look at the command reference.
[21:53] synk (~synk@ugate.dwango.co.jp) left irc: Quit: Leaving...
[21:54] <ron> http://redis.io/commands
[21:54] <ron> each command tells you its complexity.
[21:54] martincozzi (~martincoz@c-67-188-32-200.hsd1.ca.comcast.net) left irc: Quit: martincozzi
[21:54] skrul (~skrul@76.14.75.186) joined #redis.
[21:54] <dieselz29> so are you thinking I should say give me the values for day 15000, 15001, 15002 … when using hashes
[21:54] <ron> you'll notice that ZRANGE is more expensive than HMGET for the same action.
[21:55] <ron> your fields should be the days, and the values should be the hits.
[21:55] <dieselz29> i guess the difference that I was basing my idea of using sorted sets was that I could query by range in sorted sets
[21:55] <dieselz29> but i suppose if i just queried explicitly by the keys (day numbers), then that would work too
[21:56] k3VB6u4d (~k3VB6u4d@ec2-204-236-142-90.us-west-1.compute.amazonaws.com) joined #redis.
[21:57] <ron> range queries are good if you don't know the possible values within the range. since you know the values of the days, you don't really need the range.
[21:58] <dieselz29> ah gotcha. makes sense
[21:58] <ron> it's just a bit of a different way of thinking, but you'll get used to it soon enough.
[21:58] <dieselz29> it'll be even more impressive to see performance using hashes I guess. I was getting 0.1 millisecond per record write in a virtualbox on my mac with sorted sets
[21:58] <ron> when in doubt, always prefer the hash data type.
[21:59] <dieselz29> will keep that in mind
[21:59] johnsanders (~johnsande@c-69-181-21-76.hsd1.ca.comcast.net) left irc: Quit: Leaving...
[22:00] <ron> also, if you think about it, you're abusing the sorted set a bit. the 'value' should be what interests you. the score is how 'important' it is or how you want to 'order' it. if you need to change the score dynamically, there's no easy way to do it. also, you want to query according to the range on the *values* and not the *scores* which is also a bit off.
[22:00] felixge (~felixgeis@miranda/donor/theundefined) left irc: Quit: felixge
[22:01] <dieselz29> that makes sense - i just saw zrangebyscore and figured that could be helpful
[22:02] <ron> there are proper use cases for everything, obviously, but you should always try picking the right solution for the job :)
[22:02] <dieselz29> so let's see how far off base I am with my other use case: storing unique page views by using a unique identifier for each user in a set, where the key is a combination of subdomain:day
[22:04] <ron> you want to count the unique page views by users per day per page?
[22:04] <dieselz29> sorry, i didn't mean page views, i meant visitors
[22:04] <dieselz29> so the output would be a 30 day graph of unique visitors to each subdomain
[22:04] <ron> sure, makes it even better.
[22:05] <ron> yup, then the way you described it makes perfect sense to me.
[22:05] brianseeders (~BS@108-216-88-132.lightspeed.bcvloh.sbcglobal.net) left irc: Ping timeout: 248 seconds
[22:05] <dieselz29> awesome
[22:05] <ron> you can wreck it all by saying that your programming language of a choice is PHP.
[22:05] <ron> :p
[22:05] <dieselz29> .....:)
[22:05] <ron> oh god.
[22:05] <dieselz29> haha
[22:05] ron facepalms
[22:06] <ron> not enough that you're a mac user, you also 'develop' with PHP?
[22:06] <dieselz29> yea yea
[22:06] <ron> man... redis is like heaven for you ;)
[22:06] <dieselz29> does it make it worse that I'm deciding between mongo and redis?
[22:06] <ron> not at all.
[22:06] <ron> I don't see them as alternative to one another though.
[22:06] <ron> they each serve a different purpose.
[22:06] <vsmatck> yeah they're pretty different.
[22:07] <dieselz29> sure, but for this use case, I certainly could use both
[22:07] <ron> vsmatck: hello there!
[22:07] <vsmatck> ron: yo
[22:07] <dieselz29> redis would likely be better for performance, but mongo would be more flexible in the long term
[22:07] <dieselz29> in case I wanted to add more analytics on the same dataset
[22:07] <ron> dieselz29: you can use redis to keep the 30 days data (or more if you can afford it and need to query back), and use mongo to offload your data.
[22:08] <ron> I think it's a perfectly reasonable usage pattern.
[22:08] <ron> I've grown fond of mongo as well in the past few months.
[22:08] <ron> though it'll never take the place redis has in my heart ;)
[22:09] <dieselz29> planning on going one route or the other, I don't want to manage both as I'm (for all practical purposes) the only developer in the company
[22:10] synk (~synk@ugate.dwango.co.jp) joined #redis.
[22:10] <dieselz29> so ron, what are you developing with?
[22:12] <ron> Java
[22:12] ScottKevill spits.
[22:12] <vsmatck> haha
[22:12] ron grins
[22:12] <dieselz29> i would argue that for every line of php, you need to write 5-10 for the same functionality
[22:12] <dieselz29> in a startup, optimizing development time is crucial
[22:13] <dieselz29> java *certainly* has its applications, but when you need to build fast and iterate quickly, its not the right platform
[22:13] <ron> I would argue that for every line of php, you're going to drown in a sea of tears, but let's leave religious wars out of it ;)
[22:13] <ron> anyways, I gotta get ready for work.
[22:13] <ron> good luck with your adventure :)
[22:14] <dieselz29> thanks - and i really appreciate your help
[22:14] <ron> no problem.
[22:14] <ScottKevill> As long as he can part that sea of tears, the religious war will be fine.
[22:14] <ron> that's why I'm here. to assist with redis modeling and mock PHP users.
[22:14] <ron> ScottKevill ;)
[22:15] felixge (~felixgeis@91-65-245-141-dynip.superkabel.de) joined #redis.
[22:15] felixge (~felixgeis@91-65-245-141-dynip.superkabel.de) left irc: Changing host
[22:15] felixge (~felixgeis@miranda/donor/theundefined) joined #redis.
[22:15] <ScottKevill> I'm sure Java is fine if you love your horizontal scrollbar.
[22:17] <vsmatck> I watched a interesting presentation recently on Kotlin. Aims to be less verbose java on the jvm. http://www.infoq.com/presentations/The-Kotlin-Programming-Language
[22:18] grampajoe (~grampa@cpe-184-57-88-36.columbus.res.rr.com) left irc: Ping timeout: 245 seconds
[22:18] <dieselz29> bandaid on a gaping wound
[22:19] <ScottKevill> Hehe.
[22:20] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) left irc: Quit: Computer has gone to sleep.
[22:22] felixge (~felixgeis@miranda/donor/theundefined) left irc: Quit: felixge
[22:26] dieselz29 (~dieselz29@c-67-190-101-48.hsd1.co.comcast.net) left #redis.
[22:31] grampajoe (~grampa@cpe-184-57-88-36.columbus.res.rr.com) joined #redis.
[22:36] skrul (~skrul@76.14.75.186) left irc: Quit: skrul
[22:39] synk (~synk@ugate.dwango.co.jp) left irc: Quit: Leaving...
[22:40] felixge (~felixgeis@miranda/donor/theundefined) joined #redis.
[22:41] tim_smart|away (~tim@fostle.xen.prgmr.com) left irc: Quit: leaving
[22:41] tim_smart (~tim@fostle.xen.prgmr.com) joined #redis.
[22:49] lux___ (~lux@ppp-191-123.24-151.libero.it) joined #redis.
[22:56] ltbarcly (~ltbarcly@pool-71-116-105-183.snfcca.dsl-w.verizon.net) joined #redis.
[22:59] davidvip (~david@175.143.52.182) joined #redis.
[22:59] <davidvip> hi all
[23:00] <davidvip> i wanna ask if redis latest release has a install_server script for Fedora 16?
[23:07] arek_deepinit (~arek@195.189.91.22) left irc: Ping timeout: 260 seconds
[23:07] Mensa180 (467febf9@gateway/web/freenode/ip.70.127.235.249) left irc: Ping timeout: 265 seconds
[23:08] arek_deepinit (~arek@esposa.patyk.net.pl) joined #redis.
[23:09] lux___ (~lux@ppp-191-123.24-151.libero.it) left irc: Read error: Operation timed out
[23:09] merlin83 (~merlin83@unaffiliated/merlin83) left irc: Ping timeout: 276 seconds
[23:10] lux___ (~lux@ppp-191-123.24-151.libero.it) joined #redis.
[23:16] <davidvip> i wanna ask if redis latest release has a install_server script for Fedora 16?
[23:16] jrz (~jrz@d152222.upc-d.chello.nl) joined #redis.
[23:23] synk (~synk@ugate.dwango.co.jp) joined #redis.
[23:24] rittyan (~rittyan@95-26-215-143.broadband.corbina.ru) joined #redis.
[23:26] <jrz> hey
[23:27] p1d (~p1d@ns1.vonaffenfels.de) joined #redis.
[23:28] merlin83 (~merlin83@unaffiliated/merlin83) joined #redis.
[23:29] moshef (~moshef@84.94.109.182.cable.012.net.il) joined #redis.
[23:37] rb2k (~rb2k@2001:6f8:1334:0:dcf1:b992:3031:8417) joined #redis.
[23:39] davidvip (~david@175.143.52.182) left #redis ("Ex-Chat").
[23:46] jrz (~jrz@d152222.upc-d.chello.nl) left irc: Quit: http://www.damianfiligree.com
[23:48] lux___ (~lux@ppp-191-123.24-151.libero.it) left irc: Read error: Connection timed out
[23:49] lux___ (~lux@ppp-191-123.24-151.libero.it) joined #redis.
[23:53] thehodge (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) left irc: Remote host closed the connection
[23:54] thehodge_ (~thehodge@cpc5-seac20-2-0-cust310.7-2.cable.virginmedia.com) joined #redis.
[23:59] mjr_ (~mjr@cpe-66-91-117-138.hawaii.res.rr.com) left irc: Quit: mjr_
[00:00] --- Thu Nov 24 2011