SQL Saturday Austin

I was thrilled to get the email confirming that I was selected to speak at SQLSaturday #362 on January 31, 2015. 

The last time I was in Austin was 2011 for SQL Saturday 97. My wife and I flew out there and hung out with my Sister and her family while I attended the event. My wife had a great time hanging out in Texas, a first for her. And we both enjoyed the weekend with my sister. 

This time around, I am bringing my 18yr old daughter with me to hang out with my sister. Hopefully she won't corrupt her. She being my sister, and her being my daughter. 

Ever since my oldest came into the world, she has had a special affinity for her aunt, my sister. The two of them share a lot of traits and even resemble each other. Sadly, we have not lived close enough to entertain the togetherness that these two have craved for so many years. But I hope to help with that, this trip.

We will fly out Thursday before the event, and this will give my daughter all Thursday night, all day Friday and Saturday, along with most of the day Sunday to hang out with her doppelganger. I hope they have a blast, get into a ton of trouble and bond over everything they can in the short amount of time we can provide them. 

In the meantime, I will go to the speaker dinner and spend most of Saturday hanging out with my #SQLFamily at the event. The first time I was here, it was my fourth SQL Saturday event. This time it'll be like my 16th or something. I try to hit a few each year, but never more than 5, as it is difficult to justify a quantity more than that with my current situation. I would love the opportunity to go to more. But will work with what I have been given. 

The last event I attend outside my state, i took my son with me, and we made a trip out of it. At the tail end of PASS, my wife came to visit me, and we made a trip out of it as well. Austin will allow me the opportunity to bring my oldest daughter. And the very next weekend will find me in Albuquerque, having driven there with my other child. So, we are making the most out of these events, combining work and family, #sqlfamily and SQL. 

I look for to these events, to sharing my knowledge and experiences, to learning from you, and seeing my #sqlfamily. 

PASS Summit 2014 goals - crowd source solving a problem

I've not really done this before. They've usually been rattling around in my head though.
I tend to attend the conference and seek out folks that i know can add to my knowledge store or help resolve issues i am having currently.

I'm going to describe some of them here in this post, and hope that you, dear reader, can assist me with tracking them down.

I am not a BI guy, but I would like to become more of one.

One of the current projects we are working on is getting a cloud based version of our application to work well. It is the basic application we have now, only instead of getting data from an on premise sql server, its gonna reach into an azure blob storage and retrieve a json document. This rich document contains all the data we normally have, without the constraint of structure in an rdbms. But it will mimic this structure. So there will be a document describing a student, and within that, there will be names and the like. Another document within this json doc will have scores and history and other descriptive data detailing what this student has done. Think 20 tables with 10-15 fields per table, and many rows of data (or documents in this case). These json docs will grow and grow with use, as more data is added.

How do I get this data out of these json documents, and into a system where others (internal, external, application, services) can get to the data? Think for reporting. Each json doc is a single student. Maybe i need to summarize all the students from an entire region and gain insight on something they have done. I would assume that these individual rich json docs will need to be extracted to some other structure, and transformed and loaded to a storage system, be it an rdbms, or hadoop cluster, or some other magical solution. Maybe a data warehouse. Maybe a SQL Server. Maybe a MondoDB store.

What do you think it should be? How best to create this process of extracting this data and presenting it to others in a reportable fashion?

So if you are still reading this, and have an opinion on this above issue, come grab me and let's talk about it. I'd love to hear your take on this, and experience, and possible direction.

With this in mind, there are a slew of sessions that are BI related that I am going to try hard to attend this year. I have flagged some of them and hope to get some insight there, as well as with other groups like SQLCAT and the other forums and opportunities that the PASS Summit offers. I'll even sit down with vendors and spell this out in the hopes that they have a magic bullet or at least a suggestion of direction.


I suggest that you too bring issues, real issues, to the summit and attempt to get them solved. At the very least, by you talking through the issue with others, you will discover things along the way. Maybe you will solve it by yourself, or maybe you'll get put in touch with that solitary individual in the world that has already solved it and is willing to train you on the process. Or something in between. Either way, its better than sitting at your desk making it up yourself.

Get out and get your lurn on at the summit this next week and enjoy yourself.

PASS Elections 2014: my thoughts

I voted.

I think you should vote also. Go vote.

See ya at summit!

On Speaking

Back in 2004, I attended my first PASS Summit conference. I was not as integrated into the community back then, and only knew a few folks. I met some great folks on that adventure, some i still know today and some that have become quite celebrities in the community. I attended because a local user group leader encouraged me to attend, as he encouraged me to do a great many things. One of which was to speak at the local User Group. I was frightened, and exhilarated.

Having grown up in a religion that encourages its members to speak, I have had various opportunities throughout my life to stand in front of an audience and speak. I did so as a youth many times, also as a missionary, and as an adult. One in particular will always remain fresh in my mind. Usually as a youth, after speaking to the congregation, you get the 'good job' and 'atta boys' doled out by members of your congregation. They do it to make you feel better, since you probably looked terrified while in front of them. I received some of these while at church. But after church, members called me at home to congratulate me. This was when phones were not as easily accessible as today, so it seemed like an extra effort to call a child and buoy them up for a job well done. I was elated. And terrified. It meant that I had a knack for speaking, that I didnt appear to be as terrified as I felt, and that meant that I oughta continue practicing the talent, or it would become latent. So I did. I never really shied away from opportunities to speak.

Fast forward to the User Group leader who asks me to speak. I say yes. I give it a shot. I am thrilled at being selected to speak, and humbled at the task, as well as terrified. But I pull it off and get compliments. Not overflowing, but not under. A sweet spot. So I repeat at the next chance.

As my career marches on, more opportunities have presented themselves. I once spoke at a Microsoft event to a large room full of attendees, following the lesson notes they had outlined, teaching about Analysis Services. I had never used it before for real, but here I was teaching it to a crowd. I did OK. Not great, but not bad enough to recognize in myself the lack of this talent, or the subsiding of talent such that I should quit. I did OK.

I remember being asked to fly to Arizona to record some sessions for a training company. I was thrilled again. Humbled. Worried. Scared. Thrilled. Me, they picked me. I did it, and it was OK. I did good enough to be asked back on multiple occasions.

When it came time for our own community events here in my home state, I volunteered to not only speak, but to help out organize and do tasks and whatnot to help the event be accomplished. I have done this every chance I get, and have even volunteered at regional events. At this same time, I volunteer in various capacities in PASS. Each time I get to do a task, a part of me is thrilled that I was picked. I was allowed to give some of me to something bigger than me.

After a while, I notice that I may be expecting to be picked, and when this happens, I need to remind myself of the beginnings, and those feelings of humble come back. The scared and thrilled are ever present. But remembering to be humble at being picked is key. Then the level of thrill can remain high when you are picked.

I have never spoken at the PASS summit, though I have heard some members respond in surprise to this knowledge, as they coulda swore that I had. Nope. I have submitted a couple times, but have not been selected. I have been selected to SQL Saturdays and Code Camps. Once I have been asked to not speak at an event I had submitted to. Besides PASS Summit, I have been accepted to every even I have submitted. Does this make me proud? Yes. But does it mean I deserve it next time? No. I try to temper the pride with the humility of being asked to speak, because remember, that allows the thrilled feelings to be elated.


So, in a word, get over yourself. If you are not selected, you will get over it. There will be another opportunity. If you are selected to speak, awesome. Accept it with humility. Be proud you were selected. Prepare to do as good a job as you can do. Do a better job than you did last time you were given this singular honor. Be scared and thrilled. Realize that no matter how much 'celebrity' you think you are, you are just a person. Treat each opportunity like your first and react accordingly. Don't ever loose that excitement that the first infused into your being. And be scared. And be thrilled.

Can that report run faster?

We have a scenario where a report is causing us some grief. Let's call this report 'Revenue'. Let's assume that this report is shown in a 'Leadership' meeting where revenue is an important topic. Let's assume that the presenter in this meeting shows the 'Leadership' team this report by launching it in Report Manager, along with several other reports. All these reports will be used to discuss points of interest in said meeting.
So put yourself in the seat of one of the observers of said report. It is kicked off, and you watch the little spinny do its thing, while you wait for this report to materialize the data and render it visually for you to consume. You wait. And wait.

How long do you wait before you open up your device and kick off the same report to 'see if you can run it faster' than the presenter? I'm betting a few minutes, maybe 4-5. And if you do that, who else in the room is doing the same thing? Let's assume more than one individual does this, which can cause the other executions to slow, or at least causes your execution to slow.

This scenario played out a few weeks ago. Presenter kicked off report, which timed out after 8 minutes or so, but at the same time, 4 other individuals also kicked off the report. It was in the middle of these executions that I was contacted to look at said report, and see if I could make it faster. Later that day, as everyone else stopped executing it, the speed of execution dropped to a couple minutes, under 2 minutes. But during the peak of multiple executions, it was taking 8 minutes. (also, that morning, 4 times the normal quantity of reports were being generated. 4 times. as in 4000 reports, instead of 1000 reports)

Over the past few weeks, I have been digging at this report and looking for solutions to its speed issue. It seems that when normal circumstances prevail, this report can retrieve and render its data in under a minute. That seems acceptable. But under load, it takes time, and causes frustration in the team that needs to consume it.

This report is written in SSRS and hits our production reporting system that is a SQL Server 2008 R2 system that utilizes several FusionIO drives to speed things along. It is a nice machine and is much faster than our old reporting system. The data seems to be properly indexed. The queries seem to function as expected. So I jumped into caching and snapshotting to investigate alternatives to 'speeding up the report'.

Let's look at the past execution of this report to see what we are dealing with. Looking into the ExecutionLog view allows us a view into what has transpired with this particular report. I format the time to perform the tasks, and convert it to minutes, so i don't have to think too hard. I have to join to the Catalog to limit the results to the actual report that I want to see. And for sorting, let's order it by the time the report started in descending order to see the latest executions to the older ones.

1. SELECT
2.     c.[Name],
3.     (TimeDataRetrieval + TimeProcessing + TimeRendering)/60000.00 AS [Time],
4.     [RequestType],
5.     [TimeStart]
6.   from ReportServer.dbo.ExecutionLog el
7.     JOIN ReportServer.dbo.Catalog c ON c.ItemID = el.ReportID  
8.   WHERE c.[Name] = 'revenue'
9.   ORDER BY [TimeStart] desc;

This query shows me a lot of good information. I can see how many times the report was executed, how long it took, and so on. When I did this for the time frame of issues, I see exactly what I expected. A ton of executions taking a long time, and many many within the same time frames. I can expand this query to add other fields, like UserName to see who executed the report, status to see success or otherwise, among other. Use the fields you want to tell the story of the data. My story showed me that a few individuals had executed the report repeatedly and upon failure, attempted again, and again. As I mentioned, later in the day, it sped up, once everyone stopped hitting it incessantly.

I went to SSRS Report Manager to drill down to the properties of this report. On the 'Snapshot Options' tab, I enabled 'Allow report history to be created manually' and 'Store all report snapshots in history' checkbox items. In the 'Report History' tab, I saw nothing, as expected. I planned on revisiting this tab to see the progress of adding caching and snapshotting. On the 'Processing Options' tab I chose 'Cache a temporary copy of the report. Expire copy of report after a number of minutes:' and chose 240 as the minutes. I also took this opportunity to select the 'Do not timeout report' option, hoping to allow the slow report to simply continue, instead of eventually dying, which prompts others to try the same thing. And fail.
The last option that I did was to setup a cache plan on the 'Cache Refresh Options' tab. I chose 2 plans, after careful review of the executions seen from the query above. This query helped me see the times that were executed, and when a cache would be most profitable to this report.

With these 2 cache plans created, and the rest of the options selected, I was happy that the report would run better next week. As Monday rolled around, I was interested to see the results. So I grabbed the above query and executed it mid day Monday, only to encounter report executions. Meaning I could only see individuals that had fired off this report, and the time it took them to render. (which was fast). I was unable to see cache executions. I would have thought that there would be an execution at the times indicated in my cache plan, and I could see how long those took. But this query failed me. To the twitterverse!!!

After receiving some help from my #SQLFamily via @markvsql, @jasonhorner, @RateControl, and @tameraclark, I was able to create this query.

1. SELECT 
2.     ItemPath,
3.     (TimeDataRetrieval + TimeProcessing + TimeRendering)/60000 AS [Time],
4.     [RequestType], 
5.     [TimeStart]
6.   from ReportServer.dbo.ExecutionLog3 el
7.   WHERE [ItemPath] like '%revenue%'
8.   ORDER BY [TimeStart] desc

This uses ExecutionLog3, which has different fields in its view. Happily, some of these fields are exactly what I need. I can add into the where clause options to show 'Interactive' (ran by a human) or 'Refresh Cache' (ran by the caching mechanism) and see how long each took.

Looking at these results showed that the cache took longer than the interactives, as expected. But when you compare the 'Interactive' times to pre cache times, the results are astounding. The caching takes a chunk of time, and then subsequent executions are tremendously smaller in execution time. Meaning, let the system create the cache on its cache plan schedule, and then the humans can simply run the report, and get the cached results quickly. No more multiple executions. We went down from 28 executions that never seemed to complete, to 4-5 that are a lot faster.

After a few Mondays, and reviews of the resultant data, I believe I have narrowed it down to the appropriate options. I kick off the cache about a half hour before the meeting on Monday morning, and then again at noon. This allows for caching of the report data twice on a Monday, allowing the 'Leadership' meeting to have quick report executions, and cached data in the morning, and the noon cache allows for other teams to utilizes a fresher cache in their afternoon meetings. But as soon as the cache reaches 240 minutes, it expires, which reverts users back to non cached data retrieval. The rest of the week the report should run quickly in the random times it needs to run.

Point of this story is that it is important to use data to make data faster. And not a single solution is better than the other, and the appropriate solution may be something you are unfamiliar with. I did not know or utilize caching in reports much prior to this. I would have normally added indexes and tweeked the queries used to perform the data retrieval, trying to squeeze all the juice out of them that I could. But a simpler solution was to understand the use of this report, compare to other non peak time of use, and adjust appropriately.

I believe that this solution is valuable, and suits my needs, and satisfies the users of this report. Thus I share it with you in the hopes you too can learn as I did.

SQL Saturday #295 Las Vegas wrapup

(written while disconnected in the southern desert of Utah on April 8th)


This last weekend found me travelling to Las Vegas to help celebrate its first SQL Saturday ever. The team of Stacia Misner, Jason Brimhall, as well as a slew of others, were able to stand up a great event that drew a descent opening round as well as great support from local, regional and remote speakers.

As it is only a few hours (6) from my home, we decided to go as a family, spend some time in Vegas, then start a Spring Break week vacation afterwards. Various options were discussed in my family, and it was decided that we would go camping and dirt biking after an extra bit of time in Vegas. This requires us to load up anb haul our camper somewhere. We chose to camp and bike in St. George, which is partway to Las Vegas. We found a location where we could drop off our trailer and leave it for a few days while we spent time in Las Vegas.

The fateful day came when we left to make our way south. With the trailer loaded and ready we took off around 10:30 am, with 2 planned stops that would consume an hour or so, extending the trip from 6 hours to about 7 hours.

The first stop occurred as planned, and I even saved $20 I wasn't planning on. Yeah, its always nice to save money. Around 11:30 am we left that stop and headed south. We passed a town called Nephi that is an hour plus from our home. As we pass this town, there is a large gap of towns that usually means you are really on your way. The trip underway, we get excited to be going. Its about this time that I hear a small explosion from the left side of our vehicle, and am alerted to flying debris from the mirror. Having never experienced a blow out on a tire, I was surprised at how I simply knew what had happened. Blowout. Yeah. less than 200 miles from our home and at the beginning of the tirp, boom. I pull over semi gracefully and get stopped to  inspect the damage. As I exit the vehicle, I grab my nexus 5 cell phone so I can take pictures, if needed. As I exit, the phone slips from my hand and lands screen down on the pavement. It looks fine, but I can tell that it just added to the cost we had just had shoved down our throats with the blowout. Sure enough, the screen was cracked before I could even go see the damage behind us. It turns out that only 1 of the 4 trailer tires blew, which explains why it was easy to pull over after I saw the tire pieces in my mirror escaping their previously contained existence.

After several phone calls, we found a tire dealer that could send out help. When Ben arrived, he was able to pull the borked tire off our trailer. While he was there, I inquired as to his expert opinion concerning the other tires. The prognosis was rather negative, and while he returned to his shop to fix the 1 tire, we deliberated what we should do. Several options lay in front of us. Replace the single tire, replace the other partner tire on that side, replace all 4, take the trailer back home, continue on with some or all tires replaced, and so on. Ugghh. When Ben returned with a brand new tire, we were able to get it mounted easily. We had decided to return to Nephi and get all 4 tires replaced. The tires were from 2007, and though they didn't have a ton of wear and were actually had plenty of tread left, there was evidence of age and cracking visually. The time had come, we just hadn't planned on it. The tire store was amazing and got us fixed up really quickly and on the road again. 2.5 hours delayed, but on our way. Let SQL Saturday trip begin, again.

Back on the road again, with stress levels high, we got to experience them even higher as the winds blew us all over the road for the next 4 hours. At some points during the trip I had to remove my hands from the wheel and shake them out because of holding on so hard to counteract the pull of the wind gusts. Stress was a constant companion. Eventually we made it to St. George and were successfull in dropping off the trailer at storage. Phase one of the trip was complete. On to Vegas, just hours and hours delayed.

We arrived in Vegas about 20 minutes after the speaker dinner started. I went in to the dinner to grab some keys to our hotel (SQL Solutions Group had a suite of rooms rented for us and had checked in already). I intended to grab and go, but ended up hanging out a bit to talk and say hi to #SQLFAMILY members before getting my family to the hotel. After dropping the kiddos off at the hotel, my wife and I returned to the dinner and had a great time catching up with folks. It is always fun to introduce my wife to the members of my other family. The dinner that was thrown for the speakers was delicious. The gifts given to speakers were spot on for the locale. A deck of cards with the SQL Saturday Las Vegas logo, a handful of similarly logoed poker chips and a usb jump drive shaped like a poker chip. Turns out my cards have 2 5's of clubs and no 4 of clubs. But #firstworldproblems aside, I love the cards and enjoyed playing card games with my family and with them already.

At the dinner, I was able to meet and talk with several of the volunteers that would be helping out in the morning. Questions were answered, tasks discussed, and times set to meet in the morning to get it going.
The next morning I awoke late, but was at the facility a little after 7am. I grabbed the signs and went back out to the various roads to put up the directional signs to help attendees find the facility. The morning in the desert was glorious, warm, wind blowing slightly, and bright blue skies. ?I ran around with my truck and the signs and the radio blaring, pounding the sign holders into the bedrock around the area. Once complete, I returned to the facility and said more "Hi's" to folks as well as started doing various volunteer tasks. The morning got going with a couple of hiccups, mainly the printing of SpeedPasses had some issues. The rooms were up on the third floor and we found attendees had an issue initially getting around and finding the rooms and getting around. Once the layout was understood, folks had less of an issue with it. It turned out that the printed map had a typo that also lent to this confusion. It was a problem, but not a large one, and once communicated out, it was overcome. The location of the vendors was not ideal, and lent to light foot traffic to visit the vendors. This should be remedied next time if this facility is reused. The location for lunch was ideal as it was the largest room and also served as one of the session rooms. So once lunch arrived, attendees and speakers knew where it was and easily found their way there. Lunch had been catered and the catering folks did a great job keeping the supply available as the attendees flowed thru to get food. The food was good and probably one of the best lunch selections I have had at an event like this. There was a sponsor for lunch, which may have helped it be such good food as well as catered. I likey.

The facility was a bit odd, and one of the rooms had almost lounge chairs to sit in instead of the normal desk and office type chairs. The speaker stations were really high tech and had an ipad controller to select all the options for display. Initially I thought that this may prove to be a problem as each new speaker took the station, but it turned out to not be an issue, as the previous speaker had already selected the options and left it ready to go. The speaker ready room was large enough for a few speakers to get ready in, and small enough that all the speakers were unable to hang out the entire day there. This is a good thing, as it forced speakers to get out and around the event, instead of holing up in the speaker room. It was also centrally located and lent to a good flow. It's nice when the speaker room is not off in a far off location of the facility.

When the last session was completed, we all adjourned to the lunch room and got to listen to a sponsor session from Microsoft. This was a bit odd, and luckily was not the entire half hour as previously mentioned. Since it was a large room, folks gathered in the back of the room and started talking. These conversations became loud enough that the attendees actually trying to listen started turning around and giving 'that look'. Truth be told, i was one talking and received 'that look'. Suffice it to say, the location in the big room and the gathering of us all there, was not ideal for an after session. Maybe it would have been better to let folks know to convene in the big room at 5:30, after the sponsor session was over, instead of 5:00 and talking. But its never a perfect science.

After the sponsor session, the attendees were able to receive the large prizes from the sponsors. Each sponsor was given a moment to talk and give out their prizes. A lot of folks were able to receive some great prizes and they all looked happy. Once completed with the giveaways, Stacia and Jason were able to sell the event for next time, as well as PASS and the local user group. Hopefully this activity will have instilled in the local populace the excitement that we have all felt, and propel them to continue building these communities, networking with locals as well as others in the community, and help themselves and their careers.

There was an after party and though not everyone attends these, enough folks did that we were able to hang out  a bit and ultimately say our good byes to the #SQLFAMILY members we did get to see. Hugs, shakes and networking completed, we went our separate ways. Some returning to hotels, others going to the airport, and still others heading out to be entertained by the Vegas nightlife. Thus bringing to and end, the event, the learning, and the networking. Another one in the bag.

Thanks to all organizers, volunteers, speakers and attendees. Let's hope that this is the start of many more to come.

SQL Saturday #295 in Las Vegas – April 5, 2014

This weekend, SQL Saturday #295 is in Las Vegas. I am lucky enough to have been selected to present a session. I speak at 11:30 AM in DBA - Room 2 on some tips that I have used to help me with Monitoring. I have given this session before and really enjoy the topic, as well as the discussions that come from it.

As I have had to create and maintain monitoring on various machines, database servers, services and processes, I wanted to help others with some of the pain I have overcome. I got to learn powershell as a result of this process. Doing this task for my own shop has helped me learn and I enjoy sharing this with others. 

In this session, I hope to really share with you my experience, examples and tricks, and hope that you will be inspired to improve your own shop, make a change for the better, and come out of this session with something gained, be it directly related to what I talk and show or not, it matters not to me. Just that you are inspired, and become better for that inspiration.

I am bringing my entire family with me to Las Vegas for the weekend and we will start a spring break vacation after this event has concluded. We all look forward to a days of warm weather. After spending a few days in Vegas, we will go back north, and go camping and dirtbiking for the rest of the week. Returning home on friday and flying out to the next SQL Saturday even in Colorado springs. 

If you can make it out to this free event, please do. Come and learn and network and experience the #SQLFamily as we share and grow together. 

SQL Saturday #279 Phoenix wrapup

This last Saturday I had the opportunity to go from 29 degree mornings to the Arizona desert and 70 degrees during the day. Looking at it that way, it wasn't much of a choice. The weather was nice and very enjoyable. Thanks for having it. If whoever was in charge of that weather, please send some my way.

But back to the event. I took off from Utah around lunchtime and flew down to Phoenix with Randy Knight and Jason Brimhall of SQL Solutions Group fame. We had an uneventful flight down, with rough weather on both takeoff and landing. Landing in Phoenix with clothing on the bottom halves of my legs seemed like a bad idea and was remediated quickly. The evening found us at a baseball game with the rest of the speakers. Amy Lewis and crew were able to acquire tickets to the D-Backs vs Cubs game. We also got vouchers to get food in the local offerings. After standing in line for 15 minutes and waiting for a my food for a half hour I had a descent burger with some incredible garlic fries. They tasted amazing. And hours, even days later, they still tasted great. Just kidding, but they did hang around for quite a while. I enjoyed them for most of the time that they were with me. The game was disappointing if you were a DBack fan, but it was still enjoyable, with some great plays and tense moments, as well as some great time to mingle and talk with all the folks attending the event. It is always like the first day of summer camp when you get to see all your friends and catch up with them, telling stories, comparing, sharing and simply enjoying each others company. Mixing and talking to everyone, watching the game, repeating, I think that we all had a blast. With the game over, folks broke up into groups and disappeared, the evening almost over and the next day looming.

Luckily I was able to return to the hotel and not have to dig into and demos or slides, as my presentations were ready to go. I know that some were not so fortunate and went back to do more work on session prep or even more real work. Saturday was rapidly approaching and i am sure we all made it there in different way. But collect again we did, the morning of Saturday at the event. More mixing and mingling and reuniting with old friends not seen the night before occurred. The morning started with some great bfast snacks and a keynote in a large windowed room with garage door openings and massive fans. One can imagine the doors opened and the fans pushing the air around. However, since the weather was just nice and not insanely hot like Phoenix can be, the room was nice, with the blue skies glowing softly outside. 

After the keynote, folks traveled to 3 different buildings on campus to attend from 11 distinct sessions. I was up for the first session hour and had the opportunity to speak on Release Management. The session time came and I only had 2 attendees and the room monitor that was forced to attend. We waited for a bit past the start time and a few more folks came in, bringing the total up to 8. So an intimate session it was, with discussion and sharing, and me blathering on about my experiences and ideas on how to keep change from impacting us negatively in our production environments. I believe that the session was well received and I enjoyed giving it and interacting with the attendees. 

More sessions came until the lunch hour. We reunited back in the large room with the glass and glass garage doors and massive fans. Pizza and pasta and salad was on the menu as well as networking and story telling. Food was consumed and connections made. SQL was discussed and plans were made. 

Sessions resumed in the 3 locations and more learning was presumably acquired by the attendees and speakers. I got to listen to parts of 2 sessions in the afternoon and learned a few things, and need to make follow up to learn more. I love being able to have access to such great knowledge and experience. Keying into the right people and making those contacts to forge even stronger relationships through learning is what these events are about for me. Sharing and learning.

The last hour of the event found me in a room with nearly 20 other folks discussing the gloriously exciting topic of Documentation. One would not think that I could get even my closest friends tricked into coming to my session of documentation, let alone 20 strangers, but alas, this is what happened. After a lively discussion that strangely started out with an explanation of my name and ended with folks wanting to get copies of my documentation, I finished the session. 

With the session ended and the final group session beginning, the attendees that were left were energized at the chance to receive presents. We went to the keynote room and swag was given out to a ton of folks. Books, licenses and more was given away. 

Cleanup started, goodbyes were hugged out, and eventually attendees, speakers and volunteers finished and parted ways. Some to the airport, some to hotels, some back to their homes and some went to the after party at a local watering hole. More networking and stories occurred, which eventually led to more hugging and departing friends. Some of us were talked into going to a Packers Bar to witness the lovely Tim Mitchel and Kathy Kellenberger sing karaoke. After some descent performances and some, well, other performances, we were treated to our favorites sing 'Babe' to the packed bar. After that, more hugs and goodbyes. And it was over.

I enjoyed the venue. The keynote room was large and spacious, with views to the outside making it feel larger than it was. The speaker room was spacious and allowed folks to spread out and practice their craft. There was also a speaker lounge just outside the serious room where seats enough for a few more speakers to kick back and chat. One could wander in here and always find someone to talk with and scheme. The rooms were not all together, which i heard a few 'complaints' about. In quotes because they were not hearty complaints, more a discussion. But having them grouped together as they were was helpful. one could go from 1 to the next easily. Once inside the rooms, with the doors even open, it provided enough space and quiet to perform the session well without outside interruptions. Having the sponsors on the way to the sessions outside one of the buildings was a great idea, letting folks chat up the sponsors without impacting any of the rooms or hallways. And being outside was glorious. The parking was plentiful and close to the venue. The venue was close to the freeway and easily accessible. I really have no complaints about this event that would even help it be better next year. It was well done, the volunteers kept the machine running along well, and you could tell that they had had some practice with previous years, and this year it just coasted along smoothly, giving everyone a great experience.

My thanks to the sponsors and volunteers that put the event on. Thanks to the speakers that came from near and far to help with the event. And a very special thanks to the attendees for giving up their Saturday to come out and get some SQL learning on.

Cya next year!!

SQL Saturday #279 in Phoenix – March 29, 2014

This weekend, SQL Saturday #279 is in Phoenix. I am lucky enough to have been selected to present 2 sessions. I speak at 9:45 AM in DBA - Room: Room 4 on some tips that I have used to help me with Release Management. I also speak at 4:00 PM in that same room on Documentation. I have given these sessions before and really enjoy the topics, as well as the discussions that come from them.

Since we always will have to make changes to our production systems, coming up with a descent plan of attack that can be repeated to allow controlled and graceful change to occur seems like a no brainer to me. I have spent a lot of time in different organizations fine tuning processes to allow for this to occur, and it is always changing.

I believe that every person in this and many other fields of work would tell you that we lack documentation. Maybe authors don’t complain about that. But I bet a lot of others will agree. What to document, when, how, and so on. I hope in this session to give you a slew of examples, templates and ideas on how to document the things that matter most to you.

In both of these sessions, I hope to really share with you my experience, examples and tricks, and hope that you will be inspired to improve your own shop, make a change for the better, and come out of these sessions with something gained, be it directly related to what I talk and show or not, it matters not to me. Just that you are inspired, and become better for that inspiration.

It has been a long time since I was in the Phoenix area, and with the winter we have had, I look forward to a tease of warm weather for a couple days.

If you can make it out to this free event, please do. Come and learn and network and experience the #SQLFamily as we share and grow together. 

upgrade to downgrade and pull some hair out while you troubleshoot

I have a Monitoring system that houses RedGate SQL Monitor, as well as home grown monitoring solutions. It is called VOO1DBMGR1. When this system was stood up, a version of SQL Server 2012 was used that shouldn't have been. It was an Enterprise Eval version. It should have been the licensed Dev version we bought for this purpose, but without going back in time, its tough to change. 

But change it needed. I needed to upgrade it, or downgrade it, as it were. We wanted to go from Ent to Dev on sql server 2012. A testing VM was stood up to help with the process.

After several tests on testing VM that was configured for me to play with, I have installed and uninstalled and upgraded and removed SQL Server 2012 a bunch of times. After reading several blogs and articles, I found a path to upgrade (or downgrade in this case). Once the eval time expired, things stopped working like SQL Server Management Studio stopped working on box. It would error upon launch. One could continue to connect to it remotely via SSMS, but not locally. Also, when one shut of SQL Services on box, one was unable to start them again without setting the date back into the distant past and resetting the date once services started up again. This proved to be a problem.  

I discovered that once SQL Server 2012 was installed with an evaluation version, I could do a version upgrade with a properly keyed installation. We downloaded an iso from MSDN which embedded our key in it, and I was able to use this version to upgrade (downgrade actually) the version. Since it was already an Enterprise version, and we wanted Developer, it was a downgrade, but the same process could be used. So you select upgrade, it steps you thru several other steps of information before performing its operations. When done, the version has been changed. I even ran some powershell script the last time on the test system to determine that the service actually went down a couple times for a few seconds while it was upgrading.

So, it was time to perform this on VOO1DBMGR1.

I copied the same MSDN iso we used in the testing environment, and ran it. Since the key is embedded, when it comes to that screen, it’s there already, and not in ‘evaluation mode’ like the previous install was. I had the powershell script running elsewhere so I could watch the service status. As the upgrade proceeded, I saw that the service went down via the powershell script. But it wasn't for a few seconds. It simply kept reporting that it was down. When I looked at the upgrade process screen, it was simply working. Not locked. Not ‘Not Responding’. Just doing. Going. Working. But no real response from it.

I waited.

Then I looked at the windows application logs, and saw that an attempt to start the SQL Service had been attempted, and failed. The reason? ‘SQL Server evaluation period has expired.’ But that should have been taken care of with the upgrade, no? I believe so. That’s what happened in the test environment. But not here. 3 attempts so far. I looked into the ErrorLog of SQL Server itself, and it had a typical start-up sequence, but then the same error about the evaluation period expiring.

Hum.

In the past, when the service didn't start, I simply set it back to 3/1/2012. And viola, it would start up services, and then I was free to reset the date. It made me feel cheap, but I did it anyway. It’s for the greater good. So, while the upgrade process was obviously stuck in a loop, I gritted my teeth and changed the date.

I was prompted with some odd screen that said it couldn't complete an operation and would I like to retry. I said yes. It asked again. I said retry. It asked yet again, and in a fit of madness, I answered the same way. The next time it asked, I canceled the screen. At which time I was prompted with the upgrade utility, and all greens across the board, and it letting me know that it had successfully completed its activity. I beg to differ, but defer to its better judgment.

I looked for my SQL Service, and it was up and running. I logged in with SSMS (an act I was prohibited from doing for oh too long on this machine) and was successful. Once in SSMS, I was able to detect that the upgrade had actually performed its operation and the version was as expected.

I set the date back to today, to shake off those cobwebs of uncertainty. I  then turned off the SQL Service, without the date being set back to 2012. And found that the services were able to turn on and off at will now, without getting dirty.

All seemed well.

Then I tried to test something from my machine, and I encountered a slew of errors. Were these related? At first, would think so. But after some breathing exercises and several tests, along with a reboot of my machine, all was well again.

All is well.

Short version.

VOO1DBMGR1, our trusty monitoring server is now on a fully licensed and has a proper version. SSMS is once again able to run on box. Remote connections are functioning as expected. Monitoring systems are up and running. And SQL Services act well now, which will allow them to be shut down for random maintenance.

Release Management : planning meetings

After avoiding the obvious for a while now, we have decided to institute recurring planning meetings to cover only Release Management tasks. Heretofore we allowed these conversations to occur naturally, when needed, and with whom needed to be present. This usually involved someone in the development arena talking to someone that would push code changes into production. Few more were involved. And to the detriment of others, others were not always included.

The good of this method of informing of 'need to change' is that it is organic, non planned, skips meetings (everyone hates meetings) and has the randomness that is requisite at times to be agile and allow change to occur. A process was still followed, but it was executed at random times. This is a plus to those that want to keep light on their feet.

The bad is that this method forces change into a system that should and could take a bit more planning prior to execution. Regardless of how agile your development teams want to be, changing production is fraught with potential disaster. Small change or even well choreographed large change can impact a system minimally and leave no lasting scaring. However, rushed, inconsiderate change, can wreak havoc. This is the type of change that is likely to occur in a rushed or unplanned manner.

So a process was instituted at least at the inception of the change request so that several necessary tasks could flow upon knowledge of an impending change. If we stuck to the task list and performed them satisfactorily, we were usually successful. Not always. But usually. Tweeks occur as time passes, and the process is tightened.

Fast forward to today. We now realize that this method of singular knowledge sharing with a select few was detrimental to others in the organization. Some poor business analyst on the north end of the building had no idea that one of his favorite tables just suffered a drastic change and the fields he commonly referred to in his reporting were just altered inextricably, and he had no forewarning of said changes.  No one thought to let him know and his reports now suffer. Sad. But no one knew. Well, someone knew, eventually, but to the chagrin of those displaying the information to others, probably some executive in a plush conference room. Oops.

So we now have instituted a recurring meeting (uugghh, peal the skin from my face would be a better use of my time...). In this meeting, we invite many. Hopefully all. But for now, many folks. These folks were chosen for their potential interest in change to our production system. They now have the option to attend a brief meeting and hear discussion of potential changes to the production system. Here is a forum in which they can ask why, when, and why. Conversations can begin here and continue to the satisfaction of all parties. Plans will be made as to when the change will occur. Bartering as to how this change can be introduced with the least impact will ensue. Parties will be informed, knowledge shared, and life will move forward.

The changes will still involve a select few. The process to perform the change and even prepare for the change will remain similar to before. Tasks being accomplished, questions asked and answered, plans created, testing, and so on. But with this little recurring planning meeting, folks are informed. Change is much less drastic and caustic. Acceptance can begin much earlier in the process and anything needing to be tweeked to allow and accept this change can be implemented much earlier in the process. No more waiting for someone else to point out the flaw, later in a meeting, and hopefully not in front of C level folks you are trying to impress.

Start with small tweeks to your Release Management process. See how you can improve it. Add some oil here, change a gear there, and before you know it, your machine that drives and introduces change into your production topology will be so smooth you won't even hear it purring along gracefully.

Be tenacious

A Release Management tale


Last night we performed a Release that affected a production database and a website. It was a fairly simple release, and we had done the same steps previously. So with a little effort, we prepared the Release Plan which contains the steps to be performed, and executed on those after hours. Within 15 minutes, all backups and snapshots and the like were done. Within a few more minutes, all new code had been successfully pushed out. Testing ensued and the Release was labeled a success.

We all finished up our tasks, our compares, our post snapshots, documentations, and so on. Emails were sent, and we logged off. All was well.

Until the morning.

The users, darn them, starting using the website and noticing some issues. They complained. Those complaints reached our ears early in the morning, before most of us made it in to the office. So from comfy home office chairs, we logged in and started looking around. Email was the form of communication initially, but this became burdensome to await for responses, and a chat room was opened up in our internal IM product so we could talk more freely.

Initially, there were members of the troubleshooting team that wanted action. Something is broke and its only natural to want to fix it as quickly as possible. Especially since users were using it and seeing the issues. Its different at night when no one is online. Less pressure then. But now, in the morning time, people are anxious and that transfers rather quickly to the rest of us.

I had to say no. We are not just going to roll back. Just be patient.

Once we all gathered and started troubleshooting, we could dig into the why. What was happening. What we thought was happening. Reading logs. Watching processes. Watching memory. And so on. At one point we even said that it was not the database. And it was suggested that I could go back to my normal tasks. But I stuck around. I didn't feel confident that we knew what was going on, and even though I could show that the database was not under any duress, I stuck it out. I kept working on it. I helped, we all helped. Others were brought in to the mix and their ideas were considered.

Fast forward. We still do not know what is happening, except that the IIS server will get a lot of memory pressure, the site will cease to function, and once it all blows up, things start over, and the site seems to work. We see this over and over. Users are in there. We are in there. All of us contributing, but there is still no smoking gun.

So I open Profiler and limit it to the particular database and web server that is having an issue. We capture everything that is happening on the db, which is a lot, and just cross our fingers. After a few more iterations of the memory consumption and release, I notice a repeating query in the profiler, just as all hell breaks loose. Its the last statement, seemingly, that was trying to execute. I grab it as is, and attempt to run it raw. It gives a divide by zero error.

Divide by Zero!

What is this query doing? Does anyone recognize it? does it have anything to do with what we pushed last night? Is data goofed? And other relevant questions were asked. After digging a bit, sure enough, deep in a view that was altered last night, a field was being divided by, and it could be zero on occasion.

I hear a muffled 'Oops' escape the developer standing behind me. 'How did that get past testing?', he asks no one in particular. We discuss for a bit, come up with a solution, and make an alteration on the fly in production that fixes this little issue. After that, the query run raw was able to complete. And as soon as we made the change, we notice the memory consumption and explosion slow down.

It didn't cease, but it did slow.

This gave us more time. More time to look deeper. We continued to watch the Profiler results. We continued to perform tests, and we continued to see the web server work for a bit, then struggle, then use all its memory, then flush everything and continue on as if it had a goldfish sized memory. All's well now, lets go. seemingly forgetting that mere seconds ago it had used and flushed all its memory.

Another query started being the last query executed just prior to the spike in memory usage. As I captured and executed this manually, it too gave us an error. Something about a field it couldn't find in the db. Some field that looked like a valid field, yet it didn't exist. After pointing it out to the developer, he incredulously stammered something like 'where did that come from?'. Turns out that the staging environment had an extra field. This field was built in to the middle ware code that had been generated, and now was trying to do its thing against production where no such field exists.

And the web server simply crashed.

Instead of throwing an error that was helpful, or logging that it got a bad result or no result or some error, it simply kept attempting the query, letting its memory consumption expand to biblical proportions, and come crashing down. Only to try again in a few minutes, as if it had no memory of the preceding few minutes.

So now we fix it.

Now we know what is causing it. And the quickest route to fixing it is to roll back. Roll back all the changes and the site should work like it did yesterday. Not like an Alzheimer patient continually performing the same unsuccessful task. Roll back the code.

The point here is that more than half this story ago you will recall that was the suggestion. Roll it back. But that suggestion was in the heat of the moment. Something was broke. We changed something. Roll it back. If we had done this, then the 2 pieces of code that were hiding well hidden within would have never been known or fixed. Dev would have re factored the release, we would have performed it again on another day, probably tested a lot better, and found the same results. Something not working right, and we had no idea what.

So it took us a few hours. So it was frustrating. So the users were unable to use the site for a bit. With sufficient communication, we let the users know. They were placated. With some time, we dug and dug and discussed, and tried, and ultimately found the culprits. Silly little things bugs are. Scampering here and there, just out of the corners of your eyes. But havoc is being caused, until they are eradicated.

I am happy that the end result was more knowledge, time spent in the forge of troubleshooting, and an actual cause to the problem instead of a quick acting rollback, ultimately hiding the problem, but reverting us to a known state.

Its the unknown that kills you. Or if not kills you, at least puts a damper on your day.

Be patient. Be thorough. Be smart. Be tenacious.

SQL Monitoring

I use a variety of methods for monitoring my database systems. Some are home grown. Others are 3rd party tools. Some look at more than SQL Server, reaching out to services and network and beyond. Some are very specific; monitoring a collection of file to ensure that imports are occurring on a regular basis.

One of the tools that I love is RedGate's SQL Monitor. I consider it my junior DBA. It is always on and always watching my systems. It lets me know when things are going awry.

With the custom metrics, I have been able to create and monitor some things that are near and dear to my heart, but are not generic enough that an alert already exists. For example, I have one custom metric that collects data about replication. Its not perfect, but the goal is to let me know when we are experiencing a particularly heavy replication period, as i may need to stay on my feet and be vigilant. Most times these periods simply pass without incident, but on occasion, in retrospect, something has gone terribly awry and one of the early indicators was heavier than normal replication business. So, thus the custom alert.

All this is to share with you some valuable information before i share the funny that I found. I was tweeking one of the alerts, and went back in time a day to see the data it had collected in the Analysis portion of the SQL Monitor tool. And the graphic representation of the data seemed to be mimicking the icon of the application itself.

 

This made me giggle and i had to share it with you.

what does a DBA do?

What I feel like I do
What I really do

Just keeping the lights on

Full Backups, Log Backups and catching developers doing things

We all have our favorite ways to ensure that our databases are backed up. We may even have our favorite scripts, tools, and so on. Talking to someone that believes differently on this topic than you can be akin to discussion between religions or politics, with heated arguments seemingly making sense on both sides.

This is not what I want to discuss. I'm just going to share with you why I like what I like and why.

I love having individual backups for each job. This is more work for me, and is not automated. But I can create a single job for a single database and schedule it at an appropriate time. When this job fires off, i can ensure that it starts, performs its task, and completes, in a timely manner, without interruption from anything else. This takes a bit of doing and scheduling, but once done, I feel that my backups are the only thing going on at that moment, and can complete successfully. If an issue arises with a single backup, it can notify that it failed, and I know by the job name which database is having issues. Most likely the other databases are all fine, and not impeded by this single failure. This is harder to accomplish if you have shared jobs or maintenance plans, in my opinion.

So I spend the time creating individual jobs, determine the appropriate time, space these times out so that each can start and complete successfully. I also configure them to notify if a failure occurs. This coupled with other monitoring ensures that if a job fails, I know about it, soon. Knowing which one failed helps speed up the recovery.

So, that's how I like it for full backups. Single, measurable, simple. But for transaction logs, this is where I get lazy. I will often let all dbs fall to the same schedule (if possible) for tlog backups, and I will run a single job that cycles thru 'all' databases and performs the transaction log backup. I will schedule this appropriately on a db server, and let it fly.

On of the side effects to this method is that the global 'all' database approach to the tlog will fail if a certain database has not had its full backup performed already. A transaction log backup cannot happen until a full backup has occurred, and it will error. Since it cycles thru 'all' databases, this forces me to have taken all databases into account in the other method.

If I do my job, then all is well. For example, a new database needs to be stood up. I get the space needed for it, create it, size it, and so on. I then create a full backup job and find the time when it should execute. I then do nothing for tlogs, as they will automatically be backed up. Good to go.

But what about the time when someone else creates a database? Shouldn't they let the DBA know? Yes, they should, but sometimes, they do not. This is when the above scenario inadvertently helps me out. Developer X creates a new db, but doesn't perform any of the steps I usually do. No backup is created, no backup job is created, and so on. So, when the tlog job kicks off, it cycles thru 'all' databases, and encounters this one with no full backup, and freaks out, and fails and emails me.

This is when I know that a DB was created, but properly configured. In a way, its like the Database Server is tattling on the developers that created the DB without my knowledge. This lets me quietly go in, size the DB, configure it properly, create the appropriate maintenance jobs and processes for it, and get it going. I usually do this without fanfare, and simply get it setup correctly, and go back to my previous tasks. But now I know that this newly minted DB has had some TLC given to it, and will fit into my topology well.

How a patch for IE11 filled up my transaction log

The story

Our developers and QA group noticed a bug in our systems with regards to IE11. When the website was used with an IE11 browser, things didn't quite work as expected. After some research and testing, they determined that a patch could be applied to our web server that would fix IE11. It introduced some new code and some changes to the .Net framework on our web server.

So far, nothing affecting the database, right? Right! After all, they did some testing and saw that the effects of applying the patch were negligible to the test system. So, onward ho, we go!

The patch was applied one Saturday night a few weeks ago. Around 6:30 pm. I was at a basketball game with my family at the time, and noticed an alert from RedGate's SQLMonitor about a failed job. Odd I thought. (I had no idea that the patch had been applied at this point). Since it was a single job failure, I chose to continue with the family activity. An hour later, another one appeared. Another job failure. Odd. This is a simple job that is doing some simple validation of errors received from our web site services. Its never failed before.

Again, the game is almost over, and since it a once an hour job, I pushed it off a bit. When the game was over, instead of heading home, we headed to the office. My entire family. They love it, for about 5 minutes. But they bear it when it happens. In this case, we only spent an hour or so at the office. As I dig into the failed job, i realize that its a powershell script that is failing. It seems to be failing on the send email portion of the script. Duh. How lame. I could have swore that it had been tested and functioning at one point, but alas, it is failing. With some tweeking (not twerking) I get it to work, and execute it again manually. This time it sends mail. Yeah! Problem solved? No.

The email it sent me went on to describe that the errors received from the web site services had surpassed a threshold of acceptability. Oops. So a few hours ago, let's lie and say 2, I received an alert that should have told me that a threshold had been exceeded. Something bad was going on. But since the monitor couldn't email me, I didn't know the true depth of the issue.

A bit more digging finds me locating some 7k emails indicating an error I've never seen before. A short phone call later to a developer helps me see that this is a new error, and it might be associated with the patch that was applied to the web site (news i uncovered during my research of the situation). We do a quick cycle of the app pool and viola, the error stops happening in the frequency it was just bombarding us. Yeah!. Problem solved? No.

The Next Day

The next day several folks hit this issue and dig and dig and determine that the patch did something odd to the web site services, which in turn caused this new error to arise when our engines communicated from the outside world to our central services, attempting to synchronize their data with us. So, each one of the attempts failed, causing them to be told to go away and try again another day. In the meantime, the error didn't go away. It was just suppressed for a bit. It would raise again, causing the app pool to crash, and when the app pool was restarted, it would sometimes be good for a while, and other times start or continue spewing the error message email. Still, this seemed isolated from the database. Not affecting it, only in that engines were unable to connect. Some engines. Some of the time.

After more research and several days of looking, it was determined that we should roll the patch back from the system. Maybe that would revert us to the state we were in before. Maybe. How to test for this? Not a lot of confidence was given to several ideas, but the one that was picked was to increase the verbose logging on some of the services so that we could track down what was going wrong with said services as they attempted to connect and process. Yeah!. Problem solved? No.

This increased logging was applied to the web server, which is a VM, which has a small footprint. The increased logging caused some issues as it tried to write out the massive amount of logs for all the connections made that night. This caused the response to the engines to be something like "Hey, I know you tried to contact me, your central server, but I am borked right now. So, instead of accepting your data, why don't you reset your data next time we try to talk." Of course this message was editorialized, but that is the gist. Each of the engines that tried to connect and errored was told to 'reset' next time. A reset means, purge all data on central (which can be like a gig or 2 gig. maybe 5) and resend all said data. This reset process occurs all the time, from a lot of engines. It is a process that shouldn't be a bear on the topology. Usually. So as the web services did their job this night, with more verbose logging, we should have seen a fix, no? No!. Problem solved? No.

The next day was when we realized that the verbose logging caused bottlenecks on writing out all the logging, causing the engine's requests to time out, and send back the lovely error requesting a reset. This is the day before Thanksgiving. Which I had taken off as a vacation day. So I find out this information while off of work. I was informed that tons of resets may occur tonight, but everyone thinks it will be OK. No worries. Yeah!. Problem solved? No.

Thanksgiving Eve

That night, around 130am, after we had finished watching several movies as a family, I was headed to bed, and looked at my emails. Tons of emails. Tons of alerts. Tons of something hit the fan. I log into work to find that a database log file had filled up, grown to capacity on the drive, filled the drive and was basically sitting in a bad state. Other processes were still working. For example, replication was still sending data from this db server to the reporting system. Other requests were being serviced. But this db was in a bad state. Poor SQLMonitor was freaking out trying to keep up with the errors in the log about the tlog fulled up. It turned out that some 65 engines were performing resets. Right now. All at the same time. Some had finished, but others were still trying. Data was flying in and out of the system at a breakneck speed. Delete here, insert there. Tons of changes. Since it all needed to be replicated, it stuck around in the tlog longer and larger than it had ever before done. After some research, calming my beating heart, and maybe a little cussing under my breath, I add another tlog to the system on another drive, and see some pressure released, and processes continue to flow. I watch it and monitor it for an hour or so, until after 3am, and finally call it good. Things were working as expected, just a bit backed up. Several emails explaining the situation were sent out to the team, mainly to document what was going on, what I had found, and making sure others knew what was happening in realtime. I then decided to go to bed. Yeah!. Problem solved? No.

Around 5 am I awake and check my mail only to find out that just before that time, the replication database log file had filled up as well. It didn't fill the drive, but was 100% full, and things were halted. Fearing that I would have to rebuild replication in a few days, or worse, over the holiday, I got up, and drug my self downstairs to see what I could do with my magic DBA pixie dust. After reviewing things, looking at the state of several pieces of the puzzle, I created another log file on another drive for the distribution database. I double checked the status of the other log file, to find out that it was doing fine. Replication was backed up, and latent. But it started flowing again. It was like finding those pesky beavers had built yet again another dam across my river of data. Once the river was diverted, flow occurred. Yeah!. Problem solved? At this point, its 6am the morning of a major holiday, and I have no idea. It looked good at the moment. But would take a while to catch up. After watching a bit, I went to bed.

I slept in until after 10am on Thanksgiving morning. When I awoke, I grabbed my phone and looked, and viola, all seemed well. There was even an apology from the developer that had spearheaded the movement to rollback the patch, increase the logging, and see what happens. Well, we now know what happens. We can potentially affect systems and services to the point that they get blocked up way upstream, causing untold blockage down the river of our data, and inadvertently cause database problems.

What did I learn?

This was a perfect storm situation that I had not planned for. What can i take from this experience? What can I learn from it and do better?

• I had sized the log files and drives appropriately, at least as far as normal processing occurred and the baseline of normality allowed me to measure. But in this situation, more log file size, and a bit of a larger disk, would have been the preferred configuration. Something I need to look into. 
• I was happy that I had planned on having extra luns attached that were unused for Data and Log, in the case of an emergency. Since these already existed, and had space, I could easily add files to dbs and sit them on these drives, which are raring to go. This saved a lot of heartache and pressure. The solution was easily reached in both cases, and when things settled down to normal, the extra log files were removed, returning the drives to their emergency waiting state again. 
• I also realize that for some types of alerts, it would be better if a more noisy alert occurred. When the db log file first filled up, it was around 11pm. I noticed it at 130am. I could have fixed it a lot earlier, had i been properly notified. I need to fix this. 
• We all need to be more careful about touching our production system. Something as simple as an IE11 patch to a webserver was ultimately responsible for the log file filling up. If we could have tested this much better, in conjunction with the rest of the services, and not just assuming that this patch would affect a website, that it could reach into services that could cause errors to spew, and pile up too many errors and logging which could ultimately affect data coming in, etc. etc. If, If, If. We have to be better at seeing and planning for the If situations, and not be overly cautious that it causes us to freeze in our tracks and not let any change occur.
• As smart as we all are individually and collectively, there is still much to be learned, even about our system that we have created and believe to know intimately. 

Thoughts about the SQL PASS Board of Directors Voting 2013

Volunteering is a special kind of madness that many humans suffer. They continue to give of their time to volunteer to perform tasks and duties that they would normally not do. All this in the valiant effort to make their sphere of the world a little better by their wake of work. Some humans continue to do this over and over, with little recompense. Still others rise above the rest and want to stand out as targets. Often they promote themselves or become elected to be these targets, and  while ofttimes tripling or more their volunteer efforts, continue to give and give and give. These are they that run for the board of directors for almost any organization. They will stand out, at the head of this organization, make decisions that will be possibly helpful or hurtful to the organization as a whole or even individuals. They will continue to move forward working, doing, deciding and so on, until they are either done, get kicked out, or their terms expire. All the while, they are doing what the rest of us may be afraid to do; to work hard, in the face of obstinate obstacles, trying their darnedest to do a good/great job and leave behind them a better organization than when they entered.
I must believe that they all want the best, and they all try to do the best of their abilities, and they do what most of the population of the group would rather not do, the hard stuff, the difficult decisions, the long nights, the excessive back and forth discussions attempting to create a better place than yesterday. I must believe they do this. For the betterment of all, including themselves.

So, go and figure out who among the slate of this years volunteers you want out front doing the jobs that you don't want to do, and lets get behind them, let's support them, let's buoy them up when they need it, and even when they don't. Let's get behind them and help them help us be a better organization.

And remember, we'll do this again in a few years, when a new batch comes along, with a new set of ideas and goals, and the ones that are volunteering now have all but spent their last breath of energy trying, and trying to improve this place we all call PASS.

Permissions by user

I have a script that I have tweeked and tweeked over the years that offers me a quick view into what objects have specific permissions on them per user. I have used this oft times to audit systems and prove that things are the way they should be, or to possibly generate work when they are not.

The output of this little script is the servername, databasename, username, grantor, object type, object name, and the permissions (like grant select, or deny execute).

I have used this script in the past to generate a source and destination table, and then run compares on the two outputs to see what is different between two systems that should be identical. But this version below is just a quick and raw output.

I hope that this helps.





DECLARE @MyDBName sysname
 SET @MyDBName = '#Source'
 
 IF EXISTS (SELECT * FROM tempdb..sysobjects WHERE ID = Object_ID('tempdb..#permissions') )
  DROP TABLE #permissions
 
 CREATE TABLE #permissions
 (
  [qid] [int] IDENTITY (1,1) NOT NULL,
  [ServerName] varchar(150) NOT NULL,
  [DatabaseName] varchar(150) NOT NULL,
  [UserName] sysname NOT NULL,
  [Grantor] sysname NOT NULL,
  [ObjectType] varchar(60) null,
  [ObjectName] sysname null,
  [PermissionState] varchar(60) null,
  [PermissionName] varchar(128) null
 )
 
 
 
 set NoCount on
 
 declare
  @result int,
  @ErrorMsg varchar(500),
  @execStr nvarchar(4000),
  @Version int
 
   --cycle thru the databases in this server, and retrieve the dbnames of each
   declare GetDataCursor insensitive cursor
    for
        select
       name
         from sys.sysdatabases
 --      where name not in ('Audit', 'tempdb')
 --         and status not in ( 48, 528)
 --      where name in ('Master', 'MSDB')
       Order by name
 
   if @@Error <> 0 goto ErrorProc
   --try
 
    declare
     @DBName varchar(200)
 
    open GetDataCursor
    if @@Error <> 0 goto ErrorGetDataCursor
    while 0 = 0
    begin
     fetch next from GetDataCursor into
      @DBName
 
     if @@Error <> 0 goto ErrorGetDataCursor
     if (@@Fetch_Status = 0)
     begin
 
      Set @execStr = '
       INSERT INTO #permissions ( [ServerName],[DatabaseName],[UserName],[Grantor],[ObjectType],[ObjectName],[PermissionState],[PermissionName])
        SELECT
          @@ServerName AS [ServerName],
          --DB_NAME() AS [DatabaseName],
          ''' + @DBName + ''' AS [DatabaseName],
          u.name AS [UserName],
          u2.name AS [Grantor],
          CASE
           WHEN major_id > 0 THEN o.type_desc
           ELSE ''System Object''
          END AS [ObjectType],
          CASE
           WHEN major_id > 0 THEN o.name COLLATE DATABASE_DEFAULT
           ELSE o2.name
          END AS [ObjectName],
          dp.state_desc AS [PermissionState],
          permission_name as [PermissionName]
         FROM [' + @DBName + '].sys.database_permissions dp
          LEFT OUTER join [' + @DBName + '].sys.database_principals u on dp.grantee_principal_id = u.principal_id
          LEFT OUTER join [' + @DBName + '].sys.database_principals u2 on dp.grantor_principal_id = u2.principal_id
          LEFT OUTER JOIN [' + @DBName + '].sys.objects o ON o.object_id = dp.major_id
          LEFT OUTER JOIN master.sys.sysobjects o2 ON o2.id = dp.major_id
         WHERE class = 1
          AND dp.grantee_principal_id > 0
         ORDER BY [UserName], [ObjectType], [ObjectName], [PermissionState], [PermissionName]'
 
 --      Print @execStr
      exec @Result = sp_executesql @execStr
      if @@Error <> 0 or @Result <> 0
      begin
       set @ErrorMsg = 'Error occurred retrieving data for 1st query'
       goto ErrorGetDataCursor
      end
 
     end
     else
      break
    end
    goto SuccessGetDataCursor
 
   --except
    ErrorGetDataCursor:
    deallocate GetDataCursor
    goto ErrorProc
 
   --finally
    SuccessGetDataCursor:
    deallocate GetDataCursor
   --end
 
   SELECT * FROM #permissions
 
  --finally
   SuccessProc:
 
  --except
   ErrorProc:
   
   

SLC Code Camp 2013

Tomorrow morning I will be presenting on 'Release Management' at the Salt Lake City Code Camp. I was one of the lucky ones that was selected by the community to present a session at this event.

This is a presentation that I have tailored after the chapter I was honored to write in a SQL Server 2012 book recently. This is a topic that has been close to my heart and part of my day to day job for quite some time. It was fun to prepare this session for a not exclusively SQL Server audience. I am a bit nervous about it, as I usually am before any presentation.

I have uploaded my preso and supporting files to my public Dropbox. You can find the links here.

LINK to Presentation Files

What do you do when you hit the finish line?

Do you ever find yourself at the endgame of a task, only to not know what to do next? Having never thought that you would actually end up here? Sometimes it takes so long to get there and the journey is so arduous, that you forget the goal is not the work to accomplish the task. But to complete the task.

Let's say you have a problem that seems persistent and invasive. Its always on your mind. You try this and that to surmount it, only to be stymied each time. Each attempt results in a failure, at least with the the end goal not being reached. Maybe each iteration discovers a bit more information about the problem that helps you attack it in a different direction. Maybe each iteration shows you that what you thought would solve it did not, and you simply need to revisit the drawing board. But along the way you, either consciously or unconsciously, have relegated yourself to some form of failure. Such that you now get stuck in the mire of simply attacking the formidable wall over and over with all your might, with little to no result.

I believe that we all find ourselves here on occasion. Either by our own hand or by that of others or even the simple fact that the universe seems completely against our progress. What happens to me on these occasions varies. I often will reattack with a renewed vigor, trying to think outside the box. I have been known to assemble folks and discuss and throw down the gauntlet of challenge to these others to assist me, see it from another set of eyes, find flaws in my logic, approach or tactics. I have even given up. But I believe that the one thing I almost always do, once mired in the clutches of this inability to progress forward, I forget the end game. I forget the goal. I forget the tasks I need to accomplish after this task is complete. I see no end in sight, and become shortsighted to the point that nothing else exists but the eventual success of this problem.

So, when it miraculously occurs, the often sought after success, the end goal of accomplishment, I stand there, as if I were the last one left on the battle field, unsure of what to do next. Not even realizing that I had reached my destination, not able to enjoy the moment of success.

To this end I direct these comments. Envision the moment of success. Plan for it, work towards it, and recognize it when it actually reaches your shores. Take a moment to enjoy it. Regale yourself with whatever treats you deem appropriate. Celebrate the moment. But more importantly, envision what it will look like so that when you reach it, or it reaches you unexpectedly, you will recognize it.

This is what you worked for, and it's sad to me when I reach it, but am so in the trenches that I do not properly realize that its here. That I made it. That I accomplished it. That its finished. Over. Done.

Instead of letting the moment slip past you, plan for it and recognize it when it overcomes you. Enjoy it!

Then move onto the next success moment. How many can you rack up? Now that I've written this, I'm onto the next one. Thanks for listening and letting me express my latest success moment.