29 Oct

Redis session locking

Here is an excerpt from a NewRelic trace.
Cm_RedisSession_NewRelic
If that looks familiar, keep reading.

Cause

Hard to nail it down, but likely something to do with the way the excellent Cm_RedisSession module uses client-side locking. If you have NewRelic Pro, you might see this crop up now and again in the full transaction traces as above.

My hunch is that it only happens when end-users open a bunch of pages, in tabs, in quick succession (I know I do that!), and you get those locks on their session while the the pages are generated. Any kind of Full Page Cache might be masking this problem to some extent, making it harder to replicate reliably.  If they end up blocking each other, it would explain the traces  I’ve seen with  30+ seconds under Session::read.  Thats certainly enough to trigger a timeout  on most reverse proxies or load balancers, so this is a possible cause of those elusive 503 or 504 error you’ve been getting.

 

Solution, part 1

Add <disable_locking> to your Magento local.xml :

<session_save><![CDATA[db]]></session_save>
<redis_session>
   <host>12.34.56.78</host>
   <port>6379</port> 
   <password></password>
   <timeout>2.5</timeout>
   <persistent></persistent>
   <db>1</db>
   <compression_threshold>2048</compression_threshold>
   <compression_lib>gzip</compression_lib>
   <log_level>1</log_level>
   <max_concurrency>6</max_concurrency>
   <break_after_frontend>5</break_after_frontend>
   <break_after_adminhtml>30</break_after_adminhtml>
   <bot_lifetime>7200</bot_lifetime>
   <disable_locking>1</disable_locking>
</redis_session>

Solution, part 2

NB: Older versions of Cm_RedisSession do not have this feature.
To find out, open up app/code/community/Cm/RedisSession/Model/Session.php and look for “disable_locking”.

# grep disable_locking app/code/community/Cm/RedisSession -r
app/code/community/Cm/RedisSession/Model/Session.php:                    : ! (strlen("{$config->descend('disable_locking')}") ? (bool)"{$config->descend('disable_locking')}" : self::DEFAULT_DISABLE_LOCKING); 
  • If you see the above, it will work with no further action. This updated code is bundled as of Magento CE 1.9 or EE 1.14.
  • If you don’t see that, then the module needs to be updated.

The code is here: https://github.com/colinmollenhour/Cm_RedisSession

Thanks, Colin.

Do make sure that any changes are integrated with your version control and, of course, you’ll need to clear all Magento and PHP opcode caches for the changes to take effect.

If you don’t have the capacity to implement that, I have a couple of other options…

Fallback 1: Use Memcached for sessions.

If it’s not already available, ask your hosting provider to set it up.

 <session_save><![CDATA[memcache]]></session_save>
<session_save_path><![CDATA[tcp://127.0.0.1:11211?persistent=0&weight=2&timeout=10&retry_interval=10]]></session_save_path>

For sessions, there’s nothing really wrong with Memcached – performance is great – it’s just hard to find HA implementations like Elasticache or ObjectRocket Redis. So with a multi server setup, you might be introducing a single point of failure.

 

Fallback 2: Just use the default ‘files’ handler.

<session_save><![CDATA[files]]></session_save>

If you are on a single web server, it’s really not that bad for performance. Obviously this is not a good idea for multi-web-servers, because you’d have to rely on your load balancer’s session persistence and that’s another story.

To be clear, we are only talking about sessions. The  <cache> and/or <full_page_cache> should definitely stay in Redis.