Wondering what’s been happening with the offline problems and loss of some posts? Here’s DU’s sys admin, Marc Perkel, to explain it all.
And for those who keep popping off in the comments to posts about Windows or Mac, tell us again why Linux is the future and we’re all idiots for not seeing the light?
Here’s what’s happening.
In order to spread the load and increase capacity I was trying to move the MySQL server to another computer with the main server handling the web and the second server running the MySQL database. Even though I have 8 gigs of ram in the main server every now and then for some reason I can’t explain memory usage soars the the server quits working due to high load.
Most all applications talk to MySQL through “localhost” that resolves to IP address 127.0.0.1 port 3306. Normally moving this service is trivial. I just shut down the database on the main server, move the database to another server, start it up there. Then I set up an SSH tunnel that pipes requests from the main server to the new server by intercepting requests and transporting them over TCP. Generally this just works.
However…
The morons at MySQL decided to redefine the meaning of “localhost” and when an application uses “localhost” the client libraries force it to be redirected to talk to the Unix socket /var/lib/mysql/mysql.sock instead of the TCP port. Why they did this is crazy beyond belief but there’s no way to override it short of downloading the source code and fixing it and recompiling MySQL which is the standard answer that you get whenever you complain too much about Open Source.
I however don’t give up so easily. I found a Linux utility called “socat” (socket concatenate ?) that allowed me to create a fake Unix socket and pipe it to a TCP port. And it worked – for a while. But it wasn’t reliable. As long as I was awake and in front of the computer it worked fine. But as soon as I went to sleep it shut down. So I wake up to a mailbox full of “my web site is down”.
So – I put it back on the main server till I find a real solution.
But – this is yet another example of why open source will never fully replace paid for software. If you have a commercial product people can call up the vendor and scream at them to fix bugs. When I sold commercial software and someone called me needing something fixed or added I usually had it the same day. However in the open source world you have a lot of people who have a highly inflated sense of importance who think their software is the greatest thing that was ever written and it’s crap!
What it comes down to as far as I can tell is the VI editor that comes with Unix causes brain damage. I’m totally serious about this. The VI editor absolutely sucks. It’s years behind editors I got with DOS 20 years ago and is an example of how to make an interface that totally sucks. I have refused to learn this editor because I find it offensive. I do most all my editing with a Windows editor, Textpad, and I run Samba on the Linux boxes so that I can mount them from Windows. That’s because there are no good editors in the Linux world. Other than that I use Pico which is a great little Unix editor for quick and dirty but still isn’t up to the kind of power I had in DOS or even CP/M (Wordstar) which is a better editor than any text mode editor in the Unix world.
I’m still waiting for Linux to catch up to CP/M!!!
My theory is that when people use VI it lowers their standards as to what good software should look like and causes their minds to physically alter in a way that leads to VI syndrome leading to delusions that their little piece of shit software is the greatest program in the world and that they are just so superior to microsoft that it’s just a matter of days before Microsoft collapses and everyone accepts Linux as God.
But what the open source world doesn’t get is that Windows programs actually WORK! If you want to install a windows application you download it, click NEXT, AGREE, NEXT, NEXT, NEXT, FINISH and the program is running. In the Linux world this almost never happens and when it does you’re almost sure that something has to be wrong. In Linux you have to edit cryptic config files with poor documentation. Then you try to run the application, get an error, Google the error, and go back and edit again. After many hours you might have it working or you might have to give up.
Out here in the real world localhost = 127.0.0.1 everywhere EXCEPT in MySQL. Not only did they hard code the bizarre behavior but have no way to override it. In fact they do the opposite. Whatever you set the configuration to they override it and force you to use the Unix socket. So doing something that should be simple becomes a nightmare. And you can’t talk to them because they have VI syndrome.
A lot of opinions here seem to think there is some clear trend away from windows. I found this yearly chart, and I am struggling to see any clear trend other than a growing adoption rate of Vista, and decline of XP as people migrate to the new OS.
http://marketshare.hitslink.com/report.aspx?qprid=5
A lot of talk on the internets about the explosive growth of OSX… But looking at that chart, I don’t see anything much at all happening other than G4, G5, and a few cranky w2k users switching to Intel macs.
But if you were to listen to the blog’o’sphere hyperbole, you would think MS is in trouble, but clearly they are not.
But hey carry on with the Microsoft bashing, please don’t let the facts get in your way.
#68,
All the workstations I deploy are w2k… XP and Vista offer them nothing. My current desktop is Vista (I skipped the yearly update this year, as all my gaming has moved to the xbox, no need for a faster computer this year) I rather like it. But still no pressing need to go messing with a bunch of workstations that work perfectly as they are.
Although…
I may buy one of the high end servers (actual server computer not a desktop computer with the words server stenciled onto it) to virtualize all the the w2k infrastructure, and give myself that much more time to work on other things… or blog.
I was sold on the HP hypervisor system, but now I am in a holding pattern to see what VMWare releases for the SMB markets.
Why encrypt the data to and from the back end DB and the front end web server? Why not just point WordPress to the IP of the back end? If it’s all on a local subnet you should be fine security wise. This removes one more cog on the machine that could fail.
MySQL defaults to using a socket for localhost because it’s faster. If you’re MySQL server is on the same machine as the web server than just add skip-networking to the my.cnf and don’t use TCP.
Also if you’re MySQL server is consuming large amounts of memory try something like mytop which is like top but shows you the processlist at any given moment. Usually it’s one or two queries that were poorly written or the indexes weren’t setup correctly. Writing good queries is the best thing you can do for a MySQL server performance.
I understand blogging and using an eye catching title but this is has nothing to do with the quality of open source. I’m no open source zealot nut job but come on.
Yet another item of the ever increasing list of things that sucks in the world of Dvorak. Is there anything that doesn’t suck?
For what it’s worth, I’m actually a Linux fan. I criticize out of love, and to maybe get the attention of the MySQL folks because when you see a lot of complaints about this problem and they ignore it you have to hit them over the head about it.
If DU were the only MySQL app I’m hosting then yes I could point it at a new server. But I’m hosting some 400 web sites on this box and have hundreds of MySQL apps most of which I didn’t install and all talk to “localhost”.
For those MORONS who say RTFM – show me in the manual where I can get MySQL to talk TCP instead of to the Unix socket when configured for localhost.
#64 – pedro
And Opera has such a great following… 😉
#74
“And Opera has such a VOCAL following…;)”
There I fixed it for yah.
73: Well said. Just because it can be done the way the developers wanted it done doesn’t mean that’s the way it needs to be done.
Though I’ll admit that until Linux is a viable Gaming platform like WinXP I’m not going to touch it, and stick to task-specific OSS that works better for me than closed source stuff.
Firefox: Less bloated and faster than IE 7, not to mention more secure.
Not too sure about the rest of the stuff installed.
60: Simple, huh? Most of the commands there are single keys on a standard keyboard. Face it, VI was invented before most of those commands were standard keys, and the command system is a workaround to that problem, not a quick rapid-edit system.
#73: Anyone hosting over 400 pages from one source is a complete dolt
78: So, are you a professional site hosting admin? It’s likely that only a few draw the page hits and content level that DU does, which should be obvious from the fact that the site works perfectly 99.999999% of the time, the rest being because of an issue with an expansion attempt that MySQL didn’t like.
I think postgresql was written using Emacs.
Rather than encrypting all that DB traffic you might consider a dedicated wire (physical security).
If things die when you go to bed, consider using the “blank” screen saver – avoid using Open GL screen savers on servers.
Using open source does not eliminate the need for testing prior to making a change to a production system.
If you don’t know how to tune a MySQL database, find someone who can.
If Linux is ever destroyed it will be by fascism and you won’t be any happier..
The BRDDA is available to the first person who:
1. Built their own computer before 1978 with a soldering iron and instructions from hell, complete with a 8″ floppy drive and a real front panel
2. Coded their own OS from assembler, and wrote the assembler
3. Still codes in assembler and doesn’t use DLL’s unless absolutely necessary (darnit)
4. Can code an entire UI and robot telescope controller for a TRS-100 – a C8086 with 32K ram and no disk – that still runs today, 27 years later with not one service call or bug
5. Can quit moaning about how crappy the OSes are today. “Gee, my scripts don’t work.” “Oh, Win/Mac/Linux is so bloated.” Go buy some ram and another 300 Gb HDD
Yes, I’m cranky, I meet folks all the time who think they know it all and they’re just script kiddies. I think Marc does a fine job for what he’s paid and the tools he has to work with. Not everyone is an expert on everything. I’m just guessing, but I’ll bet Marc has a day job.
Next time you have to fend off a DDos attack while keeping a popular site up and running 24/7, let us know.
Just my 2 cents (2 new gold dollars worth)
Nomex suit mode now engaged
#66 — Uh, my memory is lacking about what? It seems that my only contribution to the thread was that Firefox is sucking , Marc has been doing this sysadmin thing for a while and Dodge never made an Impala. What did I miss?
I think I will have to disagree on just one thing:
The VI rant (actually i just use VIM, but it works for all intents and purposes)
VI and notepad (or any editor using that scheme of UI) are tailored for completely different things.
example: “copy this line 200 times”
vi – push yy (copies the line to buffer) and type 199p (inserts 199 times)
notepad – select line – copy – paste – paste – paste (hold down ctrl+v until you think it’s inserted 200 lines, then count to make sure)
it’s those automation features that makes VI good to use for editing. it’s ease of learning that makes notepad good.
PS. i could rant about all the other points, but i find this one easiest to dispute 😀
80,
Ms wont let Linux go away, its their ONLY protection.
81…YEP…
Add to that…
KNOWING that compilers SELDOM do what they say. And util you Actually make it REAL Assembler, you cant TELL how much compression you can have.
82 problem with FF, is all the BELLS and whistles WE ad to it.
Hmmm, for the duration of this thread… Ive written a machine controller in vs c#. It is precise to the sub milisecond range(the polling is so fast I don’t have to debounce). It is triggering the outputs faster than the mechanical relays can cycle.
I think I could even drive stepper motors from the software… but they would be slow (500-600 steps per second) because of mechanical relays…
So why would I want to write things in assembler, er machine code, again?
Just sayan.
hmm…. 83 replies and no one noticed that this is an april fools post?
come on! ssh tunneling a localhost connection instead of changing the ip address in the applications and bitching about it not working?
I can’t even come up with a comparable stupid real world story.
And then redirecting the socket to a port and bitching it doesn’t work 100% either? April fools day or you did follow advice by the BOfH
Heck, makes you wonder how the hosters do this….
My tip: only once go through your applications and change “localhost” as the db host to “dbhost.[whateveryourdomainis].com” and set up a working dns. Next time you have to move your db host and can’t keep the same ip address (whatever that may be) you just change the A record in the dns zone.
good grief
pj
#85, GregA So why would I want to write things in assembler, er machine code, again?
Because it has to run on a uProc without windoze or some other bloated POS OS? Because it has to control 6 servos simultaneously and 20 sensors in real time and has to be accurate to arcsecond tolerances while keeping track of the Earth’s rotation, the Moon’s lunation and the Sun’s apparent movement, plus 100,000 celestial objects while slewing a 1,000 lb. telescope at 40 degrees / sec? Because the hardware has to fit in a shoebox and work in temps from -40C to +60C and work on batteries in milspec conditions?
Just sayin’.
Just to be clear, and in case there is any misunderstanding, VI is, in fact, the greatest program that was ever written. I will challenge anyone who says differently to pistols at dawn.
#87
So what you are saying is, I need to get rid of this cheap throw away mechanical relay interface, and upgrade to the next model up that has optoisolation instead of solenoids. Since its serial driven, I will put the computer in an Altoids tin and buy that off the shelf from Parallax, and write the software in basic. Run the whole thing on a 9 volt battery. Oh damn, they already have astronomy software for the Basic stamp platform, and I want this to be a challenge.
Is there a serial interface on the iPhone?
But you got me on the stepper motors… I guess I will have to use a couple of car batteries to drive those. How did you solve that?
Was it a dobsonian or equatorial mount telescope?
Just sayan.
Interestingly enough, assuming that all 400 some odd sites have a config.php with “localhost” in the db host variable, I can see where this could be a problem. I find it funny that nobody mentioned anything about the web app configs, just went around and around about mysql socket configs. It was a mistake to begin with to allow customers to setup their own web apps with ‘localhost’ as the db server without splitting the load evenly across hardware to begin with. Dunno though, scripting the whole mess could get complicated – wouldn’t want to change any other instances of localhost in customers config files that weren’t mysql related. Ok so I have one bewilderment – if the problem is in the original lack of foresight, wherein customers were allowed (or instructed) to use localhost in their web app configs, how would using M$ software, or any other software have made this any different? Big problem, it sucks I know, but hey at least there are commands for finding and replacing text in large numbers of files. Heh don’t forget to make a backup first! As far as the memory issue goes…there were a few intelligent replies there and as I’m not a MySQL admin im gonna leave that one alone!
Get linux into the Real World PLEASE!!!!
Ubuntu is the GREATEST!!!!
VISTA is a ROAD-KILL!!
You know you’re gettin’ old when…. DOS, CP/M, Wordstar, puleeze I mean I was around then too – I got to play with one of the first IBM PC’s on the market. But things change. MySQL has it’s limits and it’s been through lots of revisions. It’s clear that many open source projects become too revised and lack a certain leadership vision. On the other hand (and way over at the other extreme) you could use MS SQL server but that’s $$$. So, you want to have somebody to shout at, PAY FOR IT! Things are as they are and whining isn’t gonna change it. The jungle is gettin wilder every year, natural selection will weed out the weaklings, adapt or die. Hey Mr. D. – I miss the old days too.
Interestingly enough, assuming that all 400 some odd sites have a config.php with “localhost” in the db host variable, I can see where this could be a problem.
Still:
perl -pi -e “s/localhost/192.168.1.50/g;” config.php
will solve that problem quick enough….
xemacs is da bomb for text editing on Unix’s
#93 Yes…indeed, hehe I edited “but hey at least there are commands for finding and replacing text in large numbers of files” a few times mentioning perl and some other simple languages…but I didn’t want to come off oh what is it…like it’s a simple problem. Anyone know what distro/SQL version the so called socket “bug” is appearing in? I’m assuming that BOTH the skip_networking and the bind_address are being changed on the “remote” db server.
Also I notice that he mentions Pico…makes me wonder if the system is BSD flavored…I haven’t installed Pico on a Linux system in years.
96. Mitch:
Aye, its a good point. After typing that I thought… actually, I wouldn’t change the any configuration files like that for 400 websites at once, not unless it was my last day on the job and I never planned to work in IT again….. If I was a wiser man I’d have chosen like you and not actually said anything… 😀
I used to love PIco, before someone taught me to use Vi…. I seem to remember it was the basis for Pine, which is a lovely email client (still use it on my linux servers for system messages etc).
Someone told that Google is using Linux and that it works fine. I don’t think they use MySQL. I think it’s Oracle. And I also think that Oracle is not free or opensource.
Why TF are you useing(or not using) VI?
In response to #81.
I built my first computer out of pin ball machine relays in 1972/1973 abd designed it myself.
I built my second computer in 1979. It was an IMSAI kit and was after they went out of business. I wrote my own operating system from the front panel switches.
In 1981 I built a single board Z80 kit to run a laser show. It could control 4 lasers simultaneously. It ran on a multitasking Forth that I also wrote myself.
I’ve written 2 computer languages. One was a Forth like compiler called Slick. The other was Pascalish called MarxMenu.
Started a computer retail business 1984-1990 and built and sold thousands of computers.
Had my own software company 1989-1994 and managed to sell a few million bucks worth of software. DOS utilities for managing Netware.
Self employed doing spam filtering and web hosting. The only real job I had in the last 25 years was working for EFF as their first full time sysadmin.
So – I’ve been at it a while. My programming skills are a little rusty. These days I want things to just work.