I’m excited to announce I’ll be speaking at PHPNW Conference, to be held October 9th, 2010 in Manchester, UK. I’ll be doing my talk, “Developing Easily Deployable PHP Applications“, which I also did in July at OSCON. I’m working on making the talk even better based upon feedback I received, and will talk more about the various tools we use and have developed to make building SugarCRM easier.

This is my first time to this conference, but judging from the lineup of talks it is not one to miss if you are in the area. The lineup has several exciting talks about PHP development from some of the biggest names in PHP. You can register now at the conference website; early bird registration ends on September 4th.

I hope to see many SugarCRM developers at the conference!

One of our developers recently upgraded MAMP to 1.9 on Snow Leopard, which includes imap support compiled into php but doesn’t include ssl. This meant he wasn’t able to access most of the basic mail accounts like Gmail.  He was kind enough to share his steps to resolve the issue, in the hopes of saving other developers some frustration:

Steps

1.  Download and install the c-client imap library into a directory of your choice, build the headers.
2.  Download 5.3.2 source form php, extract, and navigate to ext/imap
3.  Run phpize
4.  Configure, make, and copy the new objects to your php extension directory.  Configure command should be close to the following:

./configure –with-imap=/usr/local/imap-2007 –with-kerberos –with-imap-ssl=/usr/

5.  Restart apache verify from phpinfo

Troubleshooting

*With pcre during make process:

Copy the pcre.h header from your php download into /usr/include/php/ext/pcre if you get errors like “/usr/include/php/ext/pcre/php_pcre.h:44: error: expected specifier-qualifier-list before ‘pcre’
make: *** [php_imap.lo] Error 1″

sudo cp cd ~/php-5.3.2/ext/pcre/pcrelib/pcre.h /usr/include/php/ext/pcre/

*Wrong architecture

MAMP 1.8/1.9 is still a 32bit application so copy paste the following into your terminal before the configure command if you get a “wrong architecture on imap.so” when restarting apache in your error log.  Redo the build.

MACOSX_DEPLOYMENT_TARGET=10.6
CFLAGS=”-arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp”
CCFLAGS=”-arch i386 -arch x86_64 -g -Os -pipe”
CXXFLAGS=”-arch i386 -arch x86_64 -g -Os -pipe”
LDFLAGS=”-arch i386 -arch x86_64 -bind_at_load”
export CFLAGS CXXFLAGS LDFLAGS CCFLAGS MACOSX_DEPLOYMENT_TARGET

Now I’ll fulfill my earlier promise to talk about Sugar MMM (MeterMaid Maker).  I like to think this tool pulls everything together to enable minimal effort load testing.  Need records? Fill up your database with Tidbit.  JMeter’s GUI slowing you down?  Whip up some MeterMaid scripts to do what you need to do.  But I hear you complaining.  ‘I don’t want to write XML scripts.  I just want to type a simple command or click a button, and simulate what my users are going to do’.  Rejoice, for Sugar MMM is here to answer your prayers.

Setting up Sugar MMM requires that you edit the xml_strings.php file (located in the metadata folder), to tell it your Sugar login info, where MeterMaid is, and where the GUI version can read and write files.  If you’re concerned about a particular module’s performance, the command-line interface is what you want.  Observe:

php sugarmmm_cli.php –module=Accounts

Your MeterMaid directory will now contain a JMeter script to test out the Accounts module.  It will log into Sugar, open the Accounts module listview, open a record’s detail view, create relationships, and so on.  Details can be found on the Sugar MMM github wiki or README file.

If you’re interested in all the modules, or you have some crazy custom MeterMaid script you want to use, check out the GUI version.  In your browser, go to:

http://localhost/<Your Sugar Directory>/SugarMMM

Using this simple web interface, you can include any number of modules and/or custom MeterMaid scripts into the final result.  Then, run MeterMaid on the combined XML file, and you have an uber-script capable of load testing your setup until it begs for mercy.

Sugar MMM can be obtained here.

Today is my last day at Sugar,  so I bid you all farewell and good luck with your SugarCRM related activities.

In this post, I’ll briefly describe the features and uses of Tidbit, and walk you through some simple use cases.  Tidbit is a simple tool to populate your Sugar instance with large amounts of data for load testing purposes.

Using a few short command line options, you can generate records with Tidbit for various Sugar modules.  Using only the load command line argument, Tidbit will create that many Accounts, and records for other modules according to the ratios defined in the install_config.php file.  Here’s a short example:

php install_cli.php -l 100 -o

The output from Tidbit tells you what’s being created:

2 Teams
1200 EmailAddresses
10 Users
100 Accounts
100 Quotes
200 ProductBundles
400 Products
2400 Calls
1600 Emails
400 Contacts
400 Leads
200 Opportunities
400 Cases
300 Bugs
800 Meetings
400 Tasks
400 Notes
100 Documents

If you want more fine-grained control over the ratio of records created, you can edit install_config.php to suit your needs.  This file contains an array called $modules, where the module names are mapped to integer values.

If you are running a Sugar installation featuring modules built using Module builder or edited using Studio, Tidbit can create records for these modules as well.  If you give the –allmodules command line argument, Tidbit automatically detects all modules and creates records for them.  If any module has customized fields from Studio, Tidbit will detect these changes and generate the appropriate type of data for the field.

I also want to mention one more feature that helps to populate the Sugar installation with a “realistic” data set.  If you use the –allrelationships command line argument, Tidbit detects any custom relationships created using Studio and relates the records it creates according to what it finds.

Tidbit has been around for a while so I’m sure many are already familiar with it.  Still, I hope this post can serve as a reminder and inform you of recently added features.

Check out Tidbit here.

We have doing some testing on a few of our customer instances that are to be upgraded to SugarCRM 6.0.1. Everything has been going very smooth, thanks to the huge amount of focus we have put into upgrade testing over the last few months. However, we ran into one bizarre issue when doing testing on Internet Explorer.

To begin with, let me give you some insight into the process we go thru to do the upgrade testing for customer instances.

1. Our On-Demand team will deploy a copy of the customer’s instance on our internal loadtest cluster.
2. We come in and clone the instance; making an indentical copy of both the file structure and the database. This is helpful to do before and after comparisons of any customizations that have been made, and makes it easier for us to re-clone the instance in case we need to go thru the upgrade process again.
3. On the instance to upgrade, we run the Sugar silent upgrader.
4. We then login to the new instance and see how it looks.

However, step 4 above didn’t work on IE; we couldn’t login. After doing lots of investigation on changes between 6.0GA and the current 6.0.1 builds and finding nothing, we tripped across this note a user posted to the PHP Manual.

And then it dawned on us. When we closed the instances, we named the new instances using the form:
<br />
http://instancename_qa601.sugartestingcloud.com/<br />

…which was the problem. So changing it to:
<br />
http://instancenameqa601.sugartestingcloud.com/<br />

…made it work correctly. A good lesson for anyone when naming their applications instances; don’t use underscores in the server name.

MeterMaid is an XML-based language that can simplify the task of writing test plans for Adobe JMeter.  It was born from a desire to simplify and clean up our internal JMeter tests.  Using MeterMaid, you can write JMeter tests directly in a text editor without learning the arcane details of JMeter’s XML format.

Below you’ll find a MeterMaid script – it’s an example of the type of script you can generate using Sugar MMM (another testing tool that I’ll discuss in a later post).  It adds a relationship from a record in the Contacts module, to a record in another random module:

<speed>
<controller>
<name>10percent</name>
<precent>100</precent>
<http>
<name>Request module popup form</name>
<domain>localhost</domain>
<path>/SugarCE-Full-6.1.0beta/</path>
<method>GET</method>
<data>
<var name=”module”>${SUBPANEL_RELATE_MODULE_g1}</var>
<var name=”action”>Popup</var>
<var name=”hide_clear_button”>true</var>
<var name=”mode”>MultiSelect</var>
<var name=”create”>true</var>
<var name=”metadata”>undefined</var>
</data>
<regex>
<name>Search field name extractor</name>
<refname>SEARCH_FIELD</refname>
<type>body</type>
<matchnum>1</matchnum>
<defaultvalue>SEARCH_FIELD_ERROR</defaultvalue>
<expression>(input type=’text’ name=’)([a-zA-Z0-9\-_]+)</expression>
<template>$2$</template>
</regex>
</http>
<http>
<name>Search</name>
<domain>localhost</domain>
<path>/SugarCE-Full-6.1.0beta/</path>
<method>POST</method>
<data>
<var name=”${SEARCH_FIELD_g2}”>${RANDOM_CHAR_g1}</var>
<var name=”module”>${SUBPANEL_RELATE_MODULE_g1}</var>
<var name=”action”>Popup</var>
<var name=”query”>true</var>
<var name=”populate_parent”>false</var>
<var name=”hide_clear_button”>true</var>
<var name=”button”>Search</var>
</data>
<regex>
<name>Found ID regex</name>
<refname>FOUND_ID</refname>
<type>body</type>
<matchnum>1</matchnum>
<defaultvalue>1</defaultvalue>
<expression>(onclick=”send_back\(‘[a-zA-Z]+’,')([a-zA-Z0-9\-]+)(‘\);”&gt;)([a-zA-Z ]+)</expression>
<template>$2$</template>
</regex>
</http>
<http>
<name>Perform save request</name>
<domain>localhost</domain>
<path>/SugarCE-Full-6.1.0beta/</path>
<method>POST</method>
<data>
<var name=”subpanel_id”>${FOUND_ID_g2}</var>
<var name=”value”>DetailView</var>
<var name=”http_method”>get</var>
<var name=”return_id”>${RECORD_NUMBER_g2}</var>
<var name=”record”>${RECORD_NUMBER_g2}</var>
<var name=”isDuplicate”>false</var>
<var name=”action”>Save2</var>
<var name=”inline”>1</var>
<var name=”select_entire_list”>0</var>
<var name=”child_field”>${SUBPANEL_RELATE_MODULE_g2}</var>
<var name=”subpanel_module_name”>${SUBPANEL_RELATE_MODULE_g4}</var>
<var name=”subpanel_field_name”>${SUBPANEL_RELATE_MODULE_g3}</var>
<var name=”refresh_page”>0</var>
<var name=”module”>Contacts</var><var name=”return_module”>Contacts</var></data>
</http>
</controller>
</speed>

The first element you’ll notice is <speed>.  This is just the standard root element of all MeterMaid documents.  Next, you’ll see that all nodes in the document are children of a <controller> node.  This translates to a JMeter throughput controller.  The effect is that  anything inside the <controller> element will only be executed on a specified percent of total run-throughs of the test plan (here, 10%).

The remainder of the document is a series of <http> nodes, corresponding to JMeter HTTP elements.  Some of the children are self-explanatory such as path, domain, method, and data.  There are examples of a strange syntax:

${SUBPANEL_RELATE_MODULE_g1}

This is a reference to a variable created by a JMeter regular expression extractor.  The regex extractor to create this particular variable isn’t present in the script; this is because this example is a sub-script designed to be included as part of a larger script using the <script> tag.  It won’t do anything interesting when run in isolation, but it’s still useful to illustrate MeterMaid syntax.

There are several <regex> nodes present in the example, and you can read the specific purpose of all the child nodes in MeterMaid’s README.  They correspond to the fields you would fill out in the JMeter GUI.  For example, the FOUND_ID regex extractor finds the record id of a related record from the response to performing a search in the module popup form.

I hope this example, being slightly more involved than the ones from the README or github wiki, is helpful in demonstrating some of the capabilities of MeterMaid.

Head over to the Developer Tools Forum to share your questions or comments.

MeterMaid is available here.

Greetings to all you Sugar Dev Blog readers.  This is Matt Carroll, Software Engineer intern for the summer.  I wanted to write here to share my experience at SugarCRM and talk about what I’ve been working on.

In September I’ll be heading off to grad school, so I was looking for an interesting internship to help pay the bills until then.  I met Majed at a UC Berkeley career fair.  I could tell right away that he’s a smart guy, but also with a sense of humor.  My intuition was that Sugar would be a great place to work – a mix of highly talented people and relaxed atmosphere is very appealing.  And I’m happy to say, my initial impression has proved correct so far.

At the beginning, I worked on Tidbit.  For those who don’t know, Tidbit is a data generation tool for Sugar that populates the database with large quantities of bogus records for load testing.  The existing code was functional but some features were broken, and I was tasked with adding some new features as well.  I had to hunt down and fix some bugs regarding email address generation, which were affecting all the generated records.  Later on, I added command line options to automatically detect modules and relationships, and generate records for those as well.  This could come in handy if somebody is load testing a Sugar installation that uses custom built modules.

Next, I started working on what would eventually be dubbed Sugar MMM.  Trampus came up with MeterMaid, an XML based language for writing JMeter tests.  Sugar MMM pulls information from an existing Sugar installation and generates MeterMaid tests that simulate a typical user’s activity – loading listviews, detail views, creating records and relationships, and so forth.  It was a bit of a challenge to instruct JMeter to do what a Sugar user does – particularly, coming up with and debugging regular expressions to scrape usernames and ids from the HTML response was time consuming.

Working on Tidbit and Sugar MMM was a great opportunity for me to sharpen my skills.  I had some experience with PHP but I definitely feel comfortable with the language now.  I also learned a bit about how Sugar itself works, but it’s (obviously) very large and complicated and I’m still at an early stage of understanding it.  Prior to this summer most of the code I’ve written was in an academic setting, and it’s been great to work on some useful tools for automated testing – pulling together various tools and technologies to produce something valuable to Sugar customers.

There’s a very special thing going on here – a serious but relaxed atmosphere where people get things done efficiently and enjoy doing it.  Literally everyone I’ve met at Sugar has been extremely friendly, gracious, and hospitable.   But I have to give special thanks to Majed, David, and Trampus.  These guys have taken the time to share their valuable knowledge and experience and I truly appreciate it.

P.S. – As devoted readers already know, the tools I talked about in this post were recently open-sourced.  You can check them out at http://github.com/sugarcrm.

OSCON 2010 (Open Source Convention) is in full swing this week in Portland, Oregon, and I’d like to highlight some of SugarCRM’s efforts in the open source world. First and foremost, we have two of our Sugas participating in sessions at OSCON — John Mertic is giving a talk entitled Developing Easily Deployable PHP Applications, and Nick Halsey is participating in a panel discussion on Customer Driven, Vendor Supported Interoperability. Both sessions will be held on Thursday, July 22. Our CEO, Larry Augustin, will also be in attendance, as OSCON is one of his personal favorite conventions of the year.

To celebrate OSCON, today we announced that we’re open sourcing a number of our tools for functional testing and performance testing. These tools will be made available within the next 30 days on our Developer site. These are not just testing tools for Sugar, but can also be used with other web-based applications. Stay tuned to this space for more information on how to download and use these tools, which will be distributed through GitHub.

Along the same lines, I thought this week was a good time to remind you of an announcement we made back in April, when we made public the source code repository for Sugar Community Edition. The SVN repository was made public concurrent with the first beta release of 6.0, and for the past three months, daily updates of all the latest changes that we’re making to the source code have been pushed out to SVN. We’re giving developers a chance to get ahead of the game, pick up all the latest fixes, and not have to wait for the formal release to be pushed out into the wild. For instance, we’re working on our 6.0.1 maintenance release now, but you don’t have to wait for 6.0.1 to be released as a packaged update in order to pick up fixes for bugs that are impacting you.

Speaking of Sugar 6, here are some of the new enhancements and benefits in Sugar Community Edition 6.0:

• New installations of CE include the Admin Wizard, which guides administrators through the steps to set up Sugar for their organization.
• New users will also be guided through a User Wizard to ensure they configure their settings appropriately.
• There are a number of new Studio and Module Builder enhancements in 6.0 CE, all of which were requested by our community, making customizations easier to accomplish without having to modify the source code:
  • Ability to change the properties of out-of-the-box fields (required-ness, etc.)
  • Ability to customize out-of-the-box field lengths
  • Ability to customize the size of textarea fields
  • Ability to customize Quick Create views for custom modules built in Module Builder
  • Ability to customize Select field pop-up views
  • Ability to customize module Sugar Dashlets in Studio
  • Ability to revert to default module layouts
  • Ability to set the next number for auto-increment number fields in issue-type modules
  • Ability to display layout panels as tabs
  • New Datetime field data type
• At a deeper developer level, we’ve also refactored the formatting of numbers, dates, and currencies to ensure consistency across the application.

Additionally, over 1500 bugs have been marked as Fixed in the 6.0 releases, factoring in the six beta releases and three release candidates that were made available prior to GA.  You can look for a specific bug in our Bug Tracker, or click on the links below to see the list per milestone release.

• 6.0 Beta 1
• 6.0 Beta 2
• 6.0 Beta 3
• 6.0 Beta 4
• 6.0 Beta 5
• 6.0 Beta 6
• 6.0 RC 1
• 6.0 RC 2
• 6.0 RC 3
• 6.0 GA

Recently I was talking to a friend of mine who had worked in the XML group at Microsoft. We were talking about designing and implementing a product. Being in the CRM business I never realized how much usability and customer understanding goes into your decision making. I came from a C++ and Java background where focus was on the core code and not much on customer facing elements. Having been at SugarCRM for 5+ years it is amazing how some of this has crept in without me even being aware. Now I am not saying my decision making in this sense is perfect, but now a days whenever a design decisions comes in front of me I am trying to think about how the customer would use it.

For the longest time I had considered software engineering as software engineering. You could apply the same principals across the board. In talking to my friend it hit me that we were in two totally different spaces. His decision making went into configurability, extensibility, making it better for the developer. He would have preferred to create more configuration options in order to cover every possible permutation a developer/user could ever need. I realized that if I cross a similar problem I am thinking about the user. Sometimes what I think is right and what is best for the user are inline but sometimes they aren’t. Maybe adding an extra few screens so it is not as “pure” but easier to understand is the way to go. Maybe removing some configuration options although not as flexible but less confusing is correct.

Looking at how your users will actually want to use it is something we should always have at the forefront of design.

Last week, the Operations team at SugarCRM (the people that run all the systems behind sugarcrm.com, sugarforge.org, and SugarCRM OnDemand) met up in San Jose, CA to attend Velocity and DevopsDay during what some have referred to as “Superbowl week for Operations.” We spent the week networking with experts in the industry and attending sessions and panels about performance, automation, monitoring, and optimization.

It was a pretty great experience, reminding us that there are lots of other organizations facing similar operational challenges as the ones we face, and introducing us to some new ideas about how to solve these challenges. It was also refreshing to see plenty of other people using the same tools and patterns that we use! If you’d like to find out more about our experiences, check out one of our more detailed writeups:

• by Chris Kelly, NOC Manager
• by Wyatt Walter, Operations Engineer

Both myself and our CEO Larry Augustin were in Berlin last week to speak at LinuxTag 2010. For those not familar,LinuxTag is Europe’s largest open source conference and expo, bringing professional users, decision makers, developers, beginners and the Linux community together for a 4 day event packed with well over 100 vendors and over 40 sessions.

I spoke on Wednesday afternoon, giving my “SugarCRM: Your next business application framework” talk ( which was very familiar to the same talk I gave at SugarCon this past April ). The talk was received well, and for many it was their first exposure to the how powerful and easy to use the SugarCRM framework is. I also gave an interview to RadioTux, Germany*s only Linux Radio, about developing on SugarCRM and the upcoming Sugar 6 ( the audio for the interview was unfortunately lost, however we have agreed to redo the interview at a later date ).

Larry had a keynote on Friday afternoon entitled “The Move Towards Open Source and Open Clouds“. His talk centered on the new emerging trends of the software business and cloud computing, and how open source technologies and open source development methods and leading the charge. He also gave an interview to RadioTux onsite, talking about his history in open source and how open source principles can power the entrepreneurs of the future.

I would like to personally thank the organizers of LinuxTag for the opportunity for Larry and myself to participate in the conference, and hope to able to come back again next year.

Getting web applications performing optimally is a never-ending full-time job for a lot of people, including me! I manage the OnDemand environment for thousands of customers at SugarCRM. This post will walk through a few pretty simple steps that can be taken to improve the performance of SugarCRM on a Linux server with MySQL, but they mostly apply to any PHP application running on a LAMP stack.

Tuning a web application is a type of engineering, and as Rico Mariani at Microsoft puts it:

If you are not measuring, you are not engineering.

Monitoring is the key in all of the tips that follow because if you don’t know how bad things are, you won’t know which of these fixes help you out. Depending on the details of your situation, some can actually do more harm than good so it’s important to go through these one by one, understand the implications of them, and compare you data from befor and after you make any changes.

MySQL Performance

Getting things right in MySQL is another full-time job, but there are a few quick things you can do. First of all, make sure that all of your tables are the same character set. If they are not, this can greatly slow down queries that join between tables of different character sets. Take a look in the output of

SHOW TABLE STATUS FROM sugarcrm;

and make sure that everything is the same in your Collation column. If you are using utf8_general_ci in most places, and the table my_custom is a different type, just run:

ALTER TABLE my_custom CONVERT TO CHARACTER SET utf8;

Next up is making sure that you are properly utilizing the MySQL Query Cache. This cache saves the restlts of select queries so that if the same query comes in again before changes to the table, the results can be returned much more quickly. To see the configuration for this, run:

SHOW VARIABLES LIKE "query_cache%";

If query_cache_type is set to something other than “ON” or query_cache_size is set to 0, you will want to change these! 32MB is a fine start, but depending on your workload you may need (a lot) more to get the full benefit from this:

SET GLOBAL query_cache_type = 'ON';
SET GLOBAL query_cache_size = 32000000;

This one change can make a significant difference in performance. To tune your numbers, check the output of:

SHOW STATUS LIKE "Qcache%";

After some time running with the Query Cache on, you should see hits go up and lowmem_prunes should stay pretty low with free_memory not reaching 0. If free_memory seems to run out, or lowmem_prunes continue to climb, increase the size of the query_cache_size. There will continue to be lots of inserts and not_cached due to constantly changing tables and queries other than SELECT queries, so don’t worry about those numbers.

Generally, using the InnoDB storage engine instead of the default MyISAM storage engine will help out as well, but any tables that you want to be able to do full-text search on cannot be InnoDB, and there is a lot more to take into consideration here than any quick fix. If you are using all MYISAM tables, this would be a good thing to investigate. Switching to InnoDB and tuning your MySQL storage engine based on your workload are a lot more than I can fit here today, but there is a lot of information out there about these. Head over to the MySQL Performance Blog for more information than you ever wanted to know about MySQL performance (including an excelent article on the MySQL Query Cache).

All of your MySQL STATUS values should be included in your monitoring system, as graphs of these are very helpful over time for diagnosing problems and tuning cache sizes! Also, remember that the Query Cache uses up RAM so keep an eye on that as well. In this case, it will use 32MB and as you increase the query_cache_size, you are directly increasing RAM usage.

PHP Performance

Perhaps the single biggest perfomance enhancement that you can make for SugarCRM and many other PHP applications is OpCode Caching. I’ve written about some of the complexity involving this before at SugarCRM and Caching with APC, but the simple version is that OpCode caching will improve the performance of your PHP applications. At SugarCRM, we use APC which is pretty straightforward to install. On CentOS, and other rpm/yum Linux distributions simply:

yum install php-pecl-apc

or

pecl install apc

Then, enable apc by adding it’s configuration to /etc/php.ini or /etc/php.d/apc.ini, and make sure that apc.enabled is set to 1 and apc.shm_size is set to something reasonable. For one instance of SugarCRM, the default of 32 (for 32MB) is generally plenty but if you have other PHP things running on the server, a bigger number might help.

Copy /usr/share/pear/apc.php to somewhere in your webroot and visit it in your browser, and after a bit of usage of SugarCRM, you should see the “Hits” percentage climb while the “Misses” percentage gets smaller and smaller. Your hit ratio should be higher than 98%. If the “Cache full count” number is more than a very very small percentage of your hit counter, you should increase the size of memory available to APC. On some Linux systems, shm segment size is limited to 32M so instead of increasing shm_size, you will need to increase shm_segments.

Like MySQL, these numbers should be part of your monitoring system so that you know when it’s time to change things. Also like MySQL, this directly uses RAM. This example will use 32MB, and increasing it will use more.

Client-Side Performance

Why bother serving files to clients that never (or seldomly) change? A common industy best practice is to set a Cache-Control header for static content to 10 days in the future. Tools like YSlow will let you know if you are not doing this. (They will also give you a lot of other pointers on perfromance! We have internal bugs open at SugarCRM for most of the client side performance issues that tools like YSlow have identified.)

We use apache on our application servers, and to enable these cache-control headers we added this:

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf|xml|txt)$">
Header set Cache-Control max-age=36000,public
</FilesMatch>

For SugarCRM OnDemand, this cut our number of requests-per-second in half and made a several-hundred-millisecond improvement in the time it takes to load pages from SugarCRM OnDemand.

For this one, monitor your req/s and check your PHP Applications after any changes using a tool like YSlow.

SugarCRM Specific

Lastly, a few specific changes just for SugarCRM:

• Add ‘disable_count_query’ => true to config.php. This replaces the “1-5 of 200″ with a “1-5 of 6+” and makes a big difference when you have a lot of records. This configuration setting should be a default in a future release of SugarCRM.
• Disable explicit caching in the app by setting ‘external_cache_disabled’ => true in config.php. While SugarCRM can use external caches like APC’s user cache and Memcached for some things, this shouldn’t be enabled unless you have set up the infrastructure to handle these. This configuration setting be a default in a future release of SugarCRM.
• If developer mode is enabled and you are not using it, turn this off! In SugarCRM 6.0, this is under Adminstration -> System Settings -> Advanced. This is off by default, and if it gets turned on and accidentlally left on, it causes severe performance degradation.
• Turn loging down to the minimum that you need. If your SugarCRM installation is on a NFS based document root with multiple servers, this becomes even more important. This is under Administration -> System Settings -> Logger Settings. “Error” or “Fatal” are good options here.
• Turn off tracker features you are not using. This is in Admin -> Tracker. Each enabled piece of functionality causes things to be written to the database, and will slow down page loads. Leave on only what you need for reports that you have written.

Thats a start!

With those changes, you should be off to a pretty good start at improving the performance of your installation of SugarCRM! These things help us keep SugarCRM performing quickly in the SugarCRM OnDemand environment with a pretty high ratio of active users to servers, and a recent application of these tips that I did to a site that handles 200k+ unique users per month lead to a 65% decrease in system utilization and much improved response times.

[Cross posted at ckdake.com.]

There is a lot of talk about Social CRM (SCRM) these days and many customers are waiting for their CRM vendor to add SCRM features to their releases.   SugarCRM customers however don’t have to wait; they can take their first steps into the SCRM world by leveraging the powers of Sugar Studio.

Let’s say you would want to add a twitter feed on an account, contact or lead to your SugarCRM view.  But you want to do this is a scalable way.

This post discusses how you can generate scalable social links in your Sugar instance by adding any field (standard or custom) to a generated URL in an iFrame.

As an example, I’m going to be contact module and adding a twitter search on the contact.

First let’s create a custom field for the twitter name of our contact.

Then we’re going to put this field in the edit view.  This allows us to enter the contact’s twitter name when we edit all of our contact’s contact information. This also allows us to reuse the contacts twitter name in a different Studio integration later should we choose to do so.

This will look as follows in the Edit view.

Now let’s create an iFrame using the standard twitter search URL as the generated URL, but we replace the term we’re searching with the custom field name (“Field Name”] we created earlier.  In this example we’re using the following URL:

http://search.twitter.com/search?q={whateveryouwantosearchfor}

Finally in the detailed view of the contact, we’re creating a new panel, and we’re inserting the iFrame into this panel.  Note that we’ve also check marked the new “display panels as tabs” feature.  This is a feature is new in Sugar 6.

Here’s the end result.  And this is now enabled for all contacts in your Sugar instance.  You see that the twitter search is now displayed in a separate panel.  Note that you can do the same in any Sugar Module.  You can do the same for other Social information such as facebook, LinkedIn, blogs or for any Internet service where there is a consistent URL with a name variable.  You can use any field in Sugar and replace this with the variable.

This blog will cover what occurs to the metadata files during upgrades.  The files that are involved with parsing, merging and saving the metadata files are included in the upgrade zip file under the [SugarUpgradeVersion]/modules/UpgradeWizard/SugarMerge directory.  Today, both the silentUpgrade and Upgrade Wizard code execute the SugarMerge code.  The entry point to start the metadata merge process begins with the SugarMerge class.  The logic to handle merging the various views (Edit, Detail, List, Search, QuickCreate, Subpanel) is located in the EditViewMerge class and its subclasses (DetailViewMerge, ListViewMerge, QuickCreateMerge, SearchMerge and SubpanelMerge).

Below is a class diagram of EditViewMerge and its subclasses.  I only diagrammed some of the main variables and methods used by the merge code.

EditViewMerge and Subclasses Class Diagram

SugarMerge has an internal mapping to know which EditViewMerge class or subclass to associate with the file (detailviewdefs.php, editviewdefs.php, listviewdefs.php subpaneldefs.php, quickcreatedefs.php, searchdefs.php) being parsed.   When SugarMerge is invoked, it takes the following steps:

1. Scan all custom/modules subdirectories for metadata files
2. For each metadata file found, check to see if the original metadata file (modules/[module]/metadata) and target version metadata file  ([upgrade zip file]/modules/[module]/metadata)  exists
3. If the required files are found, use the appropriate EditViewMerge class or subclass to merge the the file contents and save to custom/modules/[module]/metadata directory as well as create a backup file within the corresponding directory with the ‘.suback.php’ suffix.

Each EditViewmerge class or subclass uses four main class variables in EditViewMerge to handle the specific internals of the metadata file.

For example, a simplified version of the Accounts module editviewdefs.php looks like:

$viewdefs ['Accounts'] =  array ( 'EditView' =>  array ( 'templateMeta' => ( ...), 'panels' => (...) ) )

$viewdefs is the variable name of all the contents (this is captured by the varName variable)

‘EditView’ is the second level key after the module name to denote the view (this is captured by the viewDefs variable)

‘templateMeta’ is the key to the portion of the metadata file used to generate and load the header, footer, buttons and any additional javascript needed (this is captured by the templateMetaName variable)

‘panels’ is the key to the layout portion of the view (this is captured by the panelName variable)

Therefore, the corresponding subclasses may alter these variables so that the EditViewMerge code knows the appropriate variable names to use when accessing various portions of the metadata file.  In the same fashion, the methods listed in the EditViewMerge class contain the logic to handle the merging of the various bits of data in the files.  The ListViewMerge class overrides most of EditViewMerge’s methods since the contents of the listviewdefs.php files are quite different.

The rest of this blog will cover what to expect when the metadata merge takes place.  The list below relates to what will occur when upgrading to SugarCRM 6.0.

1) For EditViews, DetailViews and QuickCreate views we will merge field attributes that have not been customized in 5.5.x, but altered in 6.0.x. This is necessary so that we retrieve any bug fixes that may have been done as well as provide the ability to allow the studio user to modify the rows and cols attributes of the textarea field.

For example, in 5.5.x versions, we defined the textarea field’s rows and cols in the metadata file:

array( 'name'=>'description', 'displayParams'=>array('rows'=>6, 'cols'=>80), 'label'=>'LBL_DESCRIPTION' )

However, in 6.0, this has been changed to:

array( 'name'=>'description', 'label'=>'LBL_DESCRIPTION' )

As a result, the upgrade will alter the metadata definition to remove the ‘displayParams’ key/value pair since the ‘displayParams’ key/value pair was not customized in 5.5.x. If however, the user had manually modified the ‘displayParams’ key value/pair to be:

array( 'name'=>'description', 'displayParams'=>array('rows'=>10, 'cols'=>100), 'label'=>'LBL_DESCRIPTION' )

Then an upgrade will preserve this ‘displayParams’ key/value pair in the metadata file. Please see bug number 36257 for more information regarding why most of the ‘displayParams’ key/value pair definitions were removed from the metadata files in 6.0.

2) For DetailView, EditView and QuickCreate views we will append new fields that have been added to the 6.0 Out-of-the-box layouts to the bottom of the default panel. This rule applies as well to flavor conversion where the Team field will be added to the bottom of the default panel. In order to determine where exactly the new field goes, the upgrade code shall do the following:

For DetailView, EditView and QuickCreate views

a) Check to see if there is an empty cell in the last row of the default panel. If so, take the first one available (left). If it’s the right, take the right cell.

b) If no filler space is available, then add a new row to the panel and add the new field to the left space.

c) Additional new files shall be added following the steps in a & b.
For 6.0, we have added the following new OOTB fields. Here is a breakdown of the files and the new OOTB fields for the layout we added.

Module Layout New Field(s) Notes Campaigns DetailView Date Modified, Date Created Cases DetailView Date Modified, Date Created Contacts DetailView Picture Contacts EditView Picture Contracts DetailView Date Modified, Date Created Employees DetailView Picture Employees EditView Picture Leads DetailView Date Created, Website For Website, assumes Website was not added to custom DetailView file Leads EditView Website For Website, assumes that Website was not added to custom EditView file Meetings DetailView Date Modified, Date Created Projects DetailView Date Modified, Date Created Targets DetailView Date Modified, Date Created

The same should occur with flavor conversions (upgrading from CE to PRO or CE to ENT).  In addition, team field will be added to the EditView, QuickCreate and DetailView views in all modules that support team security.

For SearchView and ListView views

a) We will not alter the positions of their fields as they have specified in their metadata files prior to the upgrade

b) We will merge the field metadata except for the ‘default’ attribute values. This will prevent new fields from appearing or existing fields from being removed.

c) New fields that are defined in 6.0 will have ‘default’ attribute set to false.
3) For the ListView and SearchView layouts, a number of things have changed in 6.0 that will affect the appearance of these fields in studio. There were various bug fixes applied to filter/add fields back into the Default and Hidden lists shown as well as out-of-the-box layout changes that were made. In order for a field to appear in either the Default or Hidden list for the ListView and SearchView studio layout editor, it must meet the following criteria:

• ’studio’=>’visible’ attribute has to be set in vardefs

• In the vardefs definition of the field, the ’studio’=>false attribute cannot be set

• It cannot be an id field (user_id, team_id, modified_by_user_id, currency_id, parent_id, etc. will not appear in the lists)

• It has to be a type of db or custom field

• It cannot be the ‘deleted’ field

• If the ‘query_only’ attribute for the field is defined in subpaneldefs.php, then exclude this field from the lists

• -NOTE- If you had moved the offending fields from the hidden to the default panel prior to 5.5.x, then we will keep them in the Default list; however, if they then move the offending field back into the hidden list and then save the changes, the offending field will no longer be shown.

4) For all non-customized metadata files (i.e. there is no custom/modules/[module]/metadata/[metadata file] entry), we will just replace the existing files in modules/[module]/metadata with the out-of-the-box 6.0 file. We feel that this is consistent with the behavior of previous upgrades and is easier to implement because we would otherwise have to copy the contents of their modules/[module]/metadata directory and then create the corresponding modules/[module]/metadata directory to preserve the 5.5.x out-of-the-box metadata layouts.

5) For all customized DetailView and EditView metadata files, we will keep the layout in panel format. Otherwise for non-customized metadata files, we will just replace their metadata files as mentioned in #4. Because of this there will be some modules that may now default to the tab layout for DetailView and EditView. So if the system does not contain customized EditView or DetailView layouts for the Contacts, Accounts, Leads and Targets modules, the EditView and/or DetailView layouts for these modules will appear with the tab layout format.

6) For the special cases were ‘created_by’, ‘created_by_name’ or ‘modified_by_name’ fields were declared in the DetailView 5.5.x custom layout, we will rename these fields to ‘date_entered’ upon merging to ensure that the DetailView custom layout accurately reflects that available fields in studio. This change is needed because ‘created_by’, ‘created_by_name’, ‘modified_by_name’ do not appear in the list of fields for the modules in the studio editor. The known modules that may be affected are:

Module File Field Name Notes Campaigns detailviewdefs.php created_by_name, modified_by_name Cases detailviewdefs.php created_by_name, modified_by_name Contracts detailviewdefs.php created_by_name, modified_by_name Leads detailviewdefs.php created_by Meetings detailviewdefs.php created_by_name, modified_by_name ProspectLists detailviewdefs.php created_by_name, modified_by_name Prospects detailviewdefs.php created_by_name, modified_by_name Tasks detailviewdefs.php created_by_name, modified_by_name

7) For all existing SugarCRM dashlets, the upgrade simply replaces existing search and list definitions with those defined out-of-the-box in 6.0. This may have the affect of altering previous layouts and functionality.

Way back when we released Sugar 5.1, we changed the logger engine used in the application from log4php to our own homegrown SugarLogger. Part of the design of this logger engine was to make it extensible, so that multiple logging backends could be used. However, doing this wasn’t necessarily upgrade-safe, so most people avoid mucking with it.

Now in Sugar 6, as a part of the many changes added to make it easier to create upgrade-safe customizations, you can now add a custom SugarLogger in an upgrade safe way. All that’s needed is to implement a new class based upon the LoggerTemplate, and add in your logging code, and save that class in the custom directory. It thats simple!

One such example logger you may create is one to output debugging code to FirePHP. For those not familiar with it, FirePHP is a plugin to the popular Firebug Firefox extension that allows you to write debugging code from your PHP application directly to the Firebug console. Creating a logger backend for this in Sugar is now super simple; check out the below example:

// change the path below to the path to your FirePHP install require_once('fb.php'); class FirePHPLogger implements LoggerTemplate { /** * Constructor */ public function __construct() { // set the default logger to FirePHP if the config says to. if ( isset($GLOBALS['sugar_config']['logger']['default']) && $GLOBALS['sugar_config']['logger']['default'] == 'FirePHP' ) LoggerManager::setLogger('default','FirePHPLogger'); } /** * see LoggerTemplate::log() */ public function log( $level, $message ) { // change to a string if there is just one entry if ( is_array($message) && count($message) == 1 ) $message = array_shift($message); // map the sugar logging level to the appropriate FirePHP log level switch ($level) { case 'debug': FB::log($message); break; case 'info': FB::info($message); break; case 'deprecated': case 'warn': FB::warn($message); break; case 'error': case 'fatal': case 'security': FB::error($message); break; } } }

Now if we save this file to our custom/include/SugarLogger/ directory, and set the $sugar_config['logger']['default'] to ‘FirePHP’ in config.php, Sugar will output it’s log messages to FireBug.

Wincache is a PHP accelerator extension, developed by the IIS team at Microsoft, to help speed up PHP applications deployed on Windows stacks. It has 3 different components to it.

• An opcode cache, similar to APC or eAccelerator, which speeds up PHP scripts by removing the need to compile PHP scripts into opcodes on every request. The opcode cache also has a file cache component, which stores the scripts in memory so they don’t have to be read from disk
• A user cache, similar to APC or memcache, that provides a shared memory space for storing PHP objects and variables that can be shared across requests.
• A session handler which keeps PHP from using slow file system operations to store session data.

What this means is that those deploying on Windows stacks now have a first rate solution for PHP optimization, enabling deployment of production PHP applications on Windows.

Beginning with Sugar 6, we will provide full support for Wincache 1.1 and later; you can test this support now in the weekly beta releases of Sugar 6. We also are providing an add-on package on SugarForge to enable Sugar 5.2 and 5.5 installations to use the user cache component of Wincache 1.1 and later as an add-on package that can be installed through Module Loader.

We look forward to hearing feedback on this using this extension for your Sugar installations.

Anyone who’s been around the software industry for a while has heard the phrase “eating your own dog food”.  It is the practice of a company using its own product.  In my career, I’ve always worked for software companies that live and breath this practice.  Dogfooding is second nature to me.  “Of course we’re running our own products!”

Often, eating your own dog food is a milestone in the release process.  If your own company isn’t able to run the software successfully, how can you ask your customers to use it?

Dogfooding is even more important for major new versions of an existing product than it is for new products.  With a new product, there are no existing installations to upgrade.  Users are not entirely dependent on the software to get their jobs done.  The new product is not a vital and integral part of the employees’ day.

With upgrades, though, there are expectations of how the software is going to behave.  The users cannot afford disruptions in their day, and they don’t want to wrestle with the software after an upgrade.

It’s probably no surprise to you that here at SugarCRM, we run our business on our own product.  Our internal Sugar instance is the single longest running installation of Sugar on the planet.  Every employee at SugarCRM uses our internal Sugar instance on a daily basis. It’s been heavily customized over the years, often adding functionality for our specific business needs that were eventually redone for inclusion in the base product. This means not all of those original customizations were upgrade-safe — shame on us! – due to the upgrade-safe methods being built after the fact.

We won’t go to the Release Candidate milestone with a new major version of Sugar until we’ve been running it successfully in production ourselves for at least a couple of weeks.  This gives us time to shake out any problems encountered with the upgrade, and add fixes to the upgrade process as necessary.

I’m happy to say that we did our production upgrade to Sugar 6.0 last night.  So far, the reaction from our internal users is overwhelmingly positive.  No disruptions, and they’re already seeing the benefit of Sugar 6.0’s myriad of usability improvements.  One of our veteran sales guy said to me, “That was the easiest upgrade we’ve ever done!”

The countdown is now on to Sugar 6.0 Release Candidate 1!