Search Results

Search found 874 results on 35 pages for 'scalability'.

Page 5/35 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

• Linux/bsd tcp load balancing with 10 gigabit ethernet

  - by user37899

  Okay, I've been looking at layer 4 load balancing solutions for 10 gigabit links. I need the following properties Works at 10Gig ethernet speeds. Can support long live tcp connections. up to 1mil live tcp connections. Balancer not involved in the return path. Fault tolerant with tcp session fail over. low latency and good through put. can be scripted. Either a software or hardware solution. Can it be done? Anyone doing this?

  Read the article

• best storage option for recording live streaming video

  - by Alchemical

  We're creating a new web site that includes video chat capabilities. Wowza Server is being used for the streaming media. We would like the capability for users to record their video chats in certain circumstances. Is it feasible to record the videos to the same server running Wowza? Or performance-wise, would it make more sense to store them on another server? I understand SANs are popular as well, but I'm a little concerned about cost for those as we are on a fairly tight budget. Currently we have two servers with pretty decent RAID cnofigurations for storage, one has 14TB and the other 6TB. Mostly concerned if doing the recording on the same server as the streaming server (or web server), could significantly adversely affect that server's primary function.

  Read the article

• NFS or GFS for LVS 10 Server Setup

  - by Michael Robinson

  Currently we have a 10 servers LVS hosting setup. The people we hired to set it up did not anything about GFS which was our preferred Central Storage File System Solution. As we have tight time constraint, we just told them to use whatever they were familiar with which is NFS. I have since done some research and it seems that NFS is not ideal for the type of high traffic site we are hoping to build. I couldn't find much info online about the signaficance differences between the 2. As we to setup all servers again right now, should we stick with NFS or find someone who knows how to setup GFS amd go with that. We need a setup that is highly reliable and scalable as we intend. As after initial setup is done, we expect high increases in traffic and load.

  Read the article

• Cache coherence literature for big (>=16CPU) systems

  - by osgx

  Hello What books and articles can you recommend to learn basis of cache coherence problems in big SMP systems (which are NUMA and ccNUMA really) with =16 cpu sockets? Something like SGI Altix architecture analysis may be interesting. What protocols (MOESI, smth else) can scale up well?

  Read the article

• How to scale out image hosting/serving?

  - by Continuation

  I asked this question on stackoverflow and it was suggested that I try it here: I'm building a website where users can upload photos and I'd also convert uploaded photos into thumbnails. Planning ahead, if the website gets popular, how do I scale it out so that the images (both original and thumbnails) will be stored in and served from multiple servers? Maybe a cluster? Is there any open source software that would help me in this? Thanks.

  Read the article

• Optimize apache for 10K+ wordpress views a day on 2GB RAM E6500 CPU

  - by Broke artist

  I have a dedicated server with apache/php on ubuntu serving my Wordpress blog with about 10K+ pageviews a day. I have W3TC plug in installed with APC. But every now and then server stop responding or goes dead slow and i have to restart apache to get it back. Heres my config what am i doing wrong? ServerRoot "/etc/apache2" LockFile /var/lock/apache2/accept.lock PidFile ${APACHE_PID_FILE} TimeOut 40 KeepAlive on MaxKeepAliveRequests 200 KeepAliveTimeout 2 StartServers 5 MinSpareServers 5 MaxSpareServers 8 ServerLimit 80 MaxClients 80 MaxRequestsPerChild 1000 StartServers 3 MinSpareServers 3 MaxSpareServers 3 ServerLimit 80 MaxClients 80 MaxRequestsPerChild 1000 StartServers 3 MinSpareServers 3 MaxSpareServers 3 ServerLimit 80 MaxClients 80 MaxRequestsPerChild 1000 User ${APACHE_RUN_USER} Group ${APACHE_RUN_GROUP} AccessFileName .htaccess Order allow,deny Deny from all Satisfy all DefaultType text/plain HostnameLookups Off ErrorLog /var/log/apache2/error.log LogLevel error Include /etc/apache2/mods-enabled/.load Include /etc/apache2/mods-enabled/.conf Include /etc/apache2/httpd.conf Include /etc/apache2/ports.conf LogFormat "%v:%p %h %l %u %t \"%r\" %s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined LogFormat "%h %l %u %t \"%r\" %s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %s %O" common LogFormat "%{Referer}i - %U" referer LogFormat "%{User-agent}i" agent CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined Include /etc/apache2/conf.d/ Include /etc/apache2/sites-enabled/

  Read the article

• how does a web application cope with thousands of requests?

  - by netrox

  I went to a few websites and noticed that they all use AJAX technology for many tasks such as chat, messages, and so forth. They use a lot of httprequests obviously. My question is if you build a simple website using AJAX and you expected only few people per hour and then you start to have like 1,000 members logged per hour - can a single web application handle more requests per hour if you just upgrade to faster bigger servers or do you have to rewrite the code? Exactly how do you "scale" the web application?

  Read the article

• Does PHP *have* to serialize/unserialize session data between each HTTP request? Or is there a sett

  - by Pete Alvin

  I think I understand why sessions are evil but for snappy client user experience I don't want to have to re-query the database on each HTTP request. (As a comparision, Java servlets can effortlessly keep tons of session objects in memory.) Can PHP be set to do this or does it have to serialize because it runs from CGI/FastCGI and therefore by definition is a new process each time a request comes in? I will be running PHP using LAMP.

  Read the article

• What's the piece of hardware listening on Facebook's or Wikipedia's IP address?

  - by Igor Ostrovsky

  I am trying to understand how massive sites like Facebook or Wikipedia work, for my intellectual curiosity. I read about various techniques for building scalable sites, but I am still puzzled about one particular detail. The part that confuses me is that ultimately, the DNS will map the entire domain to a single IP address, or a handful of IP addresses in the case of round-robin DNS. For example, wikipedia.org has only one type-A DNS record. So, people from all over the world visiting Wikipedia have to send a request to the one IP address specified in DNS. What is the piece of hardware that listens on the IP address for a massive site, and how can it possibly handle all the load coming from the requests for users all over the world? Edit 1: Thanks for all the responses! Anycast seems like a feasible answer... Does anyone know of a way to check whether a particular IP address is anycast-routed, so that I could verify that this really is the trick used in practice by large sites? Edit 2: After more reading on the topic, it appears that anycast is not typically used for dynamic web content. Anycast is usually used for UDP (e.g., DNS lookups), or sometimes for static content. One interesting thing to note is that Facebook uses profile.ak.fbcdn.net to host static content like style sheets and javascript libraries. Each time I ping this name, I get a response from a different IP address. However, I can't tell whether this is anycast in action, or a completely different technique. Back to my original question: as far as I can tell, even a large site will have a single expensive piece of load-balancing hardware listening on its handful of public IP addresses.

  Read the article

• Is this way of using Excel 2007 Pivot table for BI scalable ?

  - by Sim

  Hi all, Background: We need to consolidate sales data across the country to do analysis Our Internet connection/IT expertise/IT investment is not quite strong, therefore full BI solution is out of question I tried several SaaS BI solution (GoodData, ZohoReports) and while they're good, they seem not to fully support what we need We're looking at 'bout 2 millions record for every 2 months My current approach Our (10) sites currently gathers data from all their branches and consolidate them into 1 Excel file with Pivot table and embed source data In HQ, I will request 10 sites to send back those Excel files periodically We will import those Excel to our MSSQL server There will be a master Excel file, that will also have the same pivot table (as those came from site Excel file), and datasource is the MSSQL server More details For testing, I currently use MSSQL 2008 Express on my laptop So far, I imported our transactions for the past 2 months and there are 2 millions+ row in 1 table in MSSQL (we just use 1 table, corresponding to our common pivot table structure). DB size is ~ 600 MB In the master Excel file, if not including the source data, it's just < 10MB. Including the source data will increase the size to 60 MB (so I supposed Office 2007 automatically zip the data ?) I try using the Pivot (drag-and-drop fields) and the performance so far is OK (my laptop specs: C2D T7200, 3GB RAM, Windows XP) So my question is : If we're looking at full year transaction (roughly 15 millions rows in MSSQL 2008 Express, 3.6 GB in size), is there any issue with that 15 million rows in 1 table in SQL Express ? Is there any performance issue with the pivot table at that time ? Can it still embed the source data ? (I google-ed but didn't find the maximum size of source data Excel 2007 can embed) Any other suggestions on how we can better do this ? Given that we can't afford the full BI solution, any light-weight/budget/SaaS BI that you can recommend ? Thanks

  Read the article

• I have to shard a mysql database. I want to start with 12 shards on 2 machines. What is the best w

  - by Tim

  All tables are InnoDb. I would rather not use mysqldump, because the shard sizes will be about 200 GB (about 700 million rows), and that will take too long. I was hoping to just stop mysql for an hour, copy the data files to a new machine, and start back up. But you can't do this with InnoDb, as some data is in the shared tablespace. Even if I have the innodb_file_per_table option set. This is not a website, but a custom application, used by tens of thousands right now, so uptime and performance are important. I suppose I could add logic into my server application to allow for gradual rebalancing / moving of a shard. Does anyone have a better idea?

  Read the article

• Server Clustering (Django, Apache, Nginx, Postgres)

  - by system-matrix

  I have a project deployed with django, Apache, Nginx and Postgres. The project has requirement of live data viewable to customers. The projects main points are: 1. Devices in field send data to server(devices are also like website users) after login. 2. There is background import process which imports the uploaded data in postgres. 3. The webusers of the system use this data and can send commands to the devices, which devices read when they login. 4. There are also background analysis routines running on the data. All the above mentioned setup and system is deployed on one amazon EC2 cloud machine. The project currently supports over 600 devices and 400 users. But as the number of devices are increasing with time the performance of the server is going down. We want to extend this project so that it can support more and more devices. My initial thinking is, We will create one more server like current one and divide the devices amongst these to servers. But Again We need a central user and device managment point though django admin. Any Ideas? What are the best possible ways to create a scalable architecture? How can I create a Postgres Cluster and Use it with Django, if possible?

  Read the article

• Designing a web application to scale

  - by Fahim Akhter

  Hi, While designing a web application facebook application to be precise. Which can spike and increase rapidly because of it vitality and is right intensive. What point should one keep in mind while designing the DB. For example what things should I leave room for if I need to shard or have a Master/Slave combination later (with memcache) Considering I use Relational Database with mySQL

  Read the article

• What would cause Memcached to Hang for 2+ seconds?

  - by Brad Dwyer

  I'm going nuts trying to scale memcached. From their site: Memcached operations are almost all O(1). Connecting to it and issuing a get or stat command should never lag. If connecting lags, you may be hitting the max connections limit. See ServerMaint for details on stats to monitor. If issuing commands lags, you can have a number of tuning problems. Most common are hardware problems, not enough RAM (swapping), network problems (bandwidth, dropped packets, half-duplex connections). On rare occasion OS bugs or memcached bugs can contribute. Well.. it is most certainly not performing like an O(1) operation for me. Under low to normal load on our site memcached response times for get and set ops are about 0.001 seconds. Not bad. But if we triple the load we get outliers that take 100x (or in rare cases 1000x!) that long. I even had one instance where it took 2.2442 seconds for memcached to store a value. Obviously this is killing our site. Here's the output of Memcached-getStats during one of the slow periods: [pid] => 18079 [uptime] => 8903 [threads] => 4 [time] => 1332795759 [pointer_size] => 32 [rusage_user_seconds] => 26 [rusage_user_microseconds] => 503872 [rusage_system_seconds] => 125 [rusage_system_microseconds] => 477008 [curr_items] => 42099 [total_items] => 422500 [limit_maxbytes] => 943718400 [curr_connections] => 84 [total_connections] => 4946 [connection_structures] => 178 [bytes] => 7259957 [cmd_get] => 1679091 [cmd_set] => 351809 [get_hits] => 1662048 [get_misses] => 17043 [evictions] => 0 [bytes_read] => 109388476 [bytes_written] => 3187646458 [version] => 1.4.13 So things that I have ruled out so far are: Hitting the max connections limit (curr_connections of 84 is well below the default of max of 1024) Swapping - the machine has 900M out of 1024M of memory dedicated to memcached on a dedicated machine. It only appears to be using about 7MB of data as per the bytes stat. How would I diagnose the other hardware problems? prstat doesn't really show a whole lot going on in terms of CPU or memory usage. Not sure how to figure out the network problems but as this is a dedicated server on the same private network as the web box I don't think it's a connectivity issue (ping is less than a millisecond between the boxes). Is there something else I'm missing here? It's driving me nuts. Edit: Also forgot to mention that I've tried both persistent and non-persistent connections with minimal-to-no impact.

  Read the article

• Scaling a video processing application on EC2?

  - by Stpn

  I am approaching the need to scale a video-processign application that runs on EC2. So far the setup is one machine: Backbonejs frontend Rails 3.2 Postgresql Resque + S3 for storage The flow of the app is as follows: 1) Request from frontend. Upload a video. 2) Storing video 3) Quering external APIs. 4) Processing / encoding videos. 5) Post to frontend. I can separate the backend and frontend without any problems, but when it comes to distributing the backend between several servers I am a bit puzzled. I can probably come up with a temporary solution (like just duplicating apps making several instances), but since I don't really have expertise in backend system administration, there can be some fundamental mistakes.. Also I would rather have something that is scalable. I wonder if anyone can give some feedback on the following plan: A) Frontend machine. Just frontend, talks to backend via REST Api of sorts. B) Backend server (BS), main database. Gets request from 1), posts to 2) saves uploads to 3) C) S3 storage. D) Server for quering APIs. Basically just a Resque workers, that post info back to 2) E) Server for video encoding. Processes videos uploaded on 3) and uploads them back. So I will have: A)frontend \ \ B)MAIN_APP/DB ----- C)S3 Storage (Files) / \ / / \ / D)ExternalAPI_queries E)Video_Processing (redundant DB) (redundant DB) All this will supposedly talk to each other via HTTP requests. My reason for this is that Video Processing part is really the most resource-intensive and I would just run barebones application that accepts requests and starts processing them. Questions: 1) In this setup I will have the main database at B) and all other servers will communicate with it via HTTP requests (and store duplicates of databases also I guess..for safety reasons). Is it the right approach or should I have 1 database that everyone connects to (how then?) 2) Is it a good idea to separate API queries from Video Processing part? Logically they are very close (processing is determined by the result of API queries), but resource-wise Video Processing is waaay more intensive. 3) what should I use to distribute calls between backend apps based on load?

  Read the article

• Using NFS for scalable PHP/MySQL web application

  - by Jeroen Moons

  Here's the situation: I have a PHP/MySQL web application that accepts user uploads (pdf files). From these pdf files' pages a preview image is made on the fly and presented to the web app's users. Some pdfs might be on the large side, most will be under 50 MB but some extreme cases could be as large as a few hundred MB. A little waiting for the preview image for large pdf files is acceptable but no more than a minute let's say. Everything is running on one server for now, but soon the app will hit the server's limit on both storage and processing power. My idea to solve the problem: To deal with this situation I had the idea of having one or more pdf processing servers as needed, and one or more file storage servers. These two types of servers are mounted to the server on which the actual app runs using NFS. The app could then use GearMan to delegate pdf processing tasks to these processing servers. The processing server can mount the storage server and read the file stored there, process it and write its output to that server. The servers I'm talking about will be amazon ec2 instances. The web app returns a link to the resulting pdf preview image on the storage server that was used which can then be used on the front end to show the image to the user. My question: I have zero experience with apps that use multiple servers, is this idea viable or is there a better way to do it? Is an NFS setup fast and reliable enough for this situation?

  Read the article

• Scalable WordPress Host for High-Volume Site?

  - by Jonathan Eunice

  I need recommendations for a scalable web host for a high volume WordPress web site. For my purposes, high-volume might be 100K-500K visitors/hour. Might think towards a 1M/hour burst rate as a "high water mark." I know WP isn't the highest-performing platform out there (ha!), but it's non-negotiable. I can do "the usual optimizations" (e.g. put static files in a CDN, run and follow the advice of performance analyzers like YSlow, etc). But it will still be WordPress, and there will be a dozen or so plugins involved. So, where to host the site? Most "what's the best WordPress host?" discussions seem to focus on lowest-cost. I need the opposite. What are the high-volume, scalable, or clustered WordPress hosts with which you've had great experiences?

  Read the article

• Multiple servers vs 1 big server performace

  - by pistacchio

  Hi to all! My team of developers has suggested a server structure for an upcoming project we are developing. Our structure is "logical", meaning that the various logical components of the application (it is a distributed one) relies on different servers. Some components are more critical than others and will be subjected to more load. Our proposal was to have 1 server per component but the hardware guys suggested to replace the various machines with a single, bigger one with virtual servers. They're gonna use Blade Servers. Now, I'm not an expert at all, but my question to the guys was: so if we need, for example, 3 2GHz CPU / 2GB RAM machines and you give me 1 machine with 3 2GHz CPUs and 6 GB of RAM it is the same? They told me it is. Is this accurate? What are the advantages or disadvantages of both the solutions? What are the generally accepted best practices? Could you point out some URL reference dealing with the problem? Thank you in advance! EDIT: Some more info. The (internet / intranet) application is already layered. We have some servers on the DMZ that will expose pages to the internet and the databases are on their own machines. What we want to split (and they want to join) are some webservers that mainly expose webservices. One is a DAL that communicates with the database layer, one is our Single Sign On / User Profile application that gets called once per page and one is a clone of what seen on the Internet to be used on our lan.

  Read the article

• AWS: Multi-region setup using single RDS instance

  - by Ion

  I'm trying to scale our web application (PHP, MySQL, memcache) in a multi-region scheme. Currently we are using a setup with two EC2 instances behind an ELB and an RDS instance, all of them in US-EAST (Virginia) region. We would like to have a presence in the EU (Ireland) region as well. This means at least a new EC2 instance there (identical to the others, serving the same application). I have copied the desired AMI, setup the new instance, setup a same ELB configuration (required for SSL termination) and configured latency-based routing in Route53. And it works as suggested. But, clients from EU have speed problems. This is due to the fact that the EU EC2 instances connect to the US-based RDS instance. As far as I know Amazon has not yet enabled RDS multi-region replication. Do you have any suggestions on how to properly speed up the whole setup while using the single RDS instance? Also, any ideas in general on how to scale things up? Ideally we would like to continue using the RDS technology for various reasons. Nevertheless, I am open to suggestions (I guess the next idea would be to host our own MySQL servers).

  Read the article

• What Sort of Server Setup Am I Likely to Need? - School A/V streaming

  - by DeathMagus

  My prior experience with servers has generally been limited to home file-sharing servers, low-traffic web-servers, and the like. This leaves me with the technical knowledge of how to set up a system, but little experience in terms of scaling said system. My current project, however, has me as the technical lead in setting up a school for online audio and video streaming. The difficulty I'm running into is that I don't quite have the experience to guess what they'll need, and they don't have the experience to tell me - so I've tried to ask as many pertinent questions about what they want to do with their server, and here's what I found out: About 1000 simultaneous users, and hoping to expand (possibly significantly) Both video and audio streaming, at obviously the highest quality possible Support for both live and playlist-based streaming. Probably only one channel, but as it's an educational opportunity, I imagine letting them have a few more wouldn't hurt. No word on whether they're locked into Windows or whether Linux is acceptable. Approximate budget - $7000. It may actually be about $2k less than this, because of a mishap with another technology firm (they ordered a $7000 DV tape deck for some reason, and now the company wants them to pay a 30% restocking fee). The tentative decisions I've already made: I'm planning on using Icecast 2 for my streaming server, fed by VLC Shoutcast encoding. Since the school already has a DMZ set up, I plan on placing the Icecast server in there, and feeding it through their intranet from a simple workstation computer in their studios. This system isn't in any way mission critical - it's an education tool (they're a media magnet school), so I figure redundancy is not worthwhile to them from a cost:benefit perspective. What I don't know is this: How powerful of a server will I need? What is likely to be my major throttle - bandwidth? How can I mitigate that? Will I need anything special for the encoding workstation other than professional video and audio capture cards and a copy of VLC? Are there any other considerations that I'm simply missing? Thanks a lot for any help - if there's more information you need, let me know and I'll tell you all I can.

  Read the article

• Sharing / replicating EBS across AWS nodes

  - by skrat

  I would like to use single EBS storage across multiple EC2 nodes (web/app servers). I've read some articles on snapshot sharing, but that doesn't suit well for what we need. We use filesystem for storing DB record attachments, so if one such attachment gets created, we need it to be immediately available to all nodes (to serve). So far only NFS seem to be viable, but it's a pain to configure and maintain. Another option could be storing those attachments on S3 instead, but that would cut us of doing any analysis on that data. This must be quite common problem when scaling in AWS, what solutions are there?

  Read the article

• Load testing nginx inside AWS

  - by andy

  I'm trying to load test nginx running on AWS. I need to try to optimise it to handle 1Gbps of inbound traffic. Currently I've got it to peak at 85Mbit/s by running nginx on an m1.large with 4 other machines hitting it by using ab with -i (for head requests) -k (keepalives) -r (ignore failed requests) -n 500000 -c 20000. I'm struggling to generate more than 85 Mbit/s traffic from 4 machines, yet when I do scp a large file I get nearly 0.25Gbit/s of traffic going over the network. Are there any tools or approaches that I could use to load test nginx that might generate more load? I'm only interested in inbound traffic, so perhaps a DoS tool could help if it chucks away responses? I'm hitting a very small (40 byte) static asset, and have peaked at handling 50K concurrent connections and getting 25k reqs/s when just using a single load generator machine.

  Read the article

• What are the typical methods used to scale up/out email storage servers?

  - by nareshov

  Hi, What I've tried: I have two email storage architectures. Old and new. Old: courier-imapds on several (18+) 1TB-storage servers. If one of them show signs of running out of disk space, we migrate a few email accounts to another server. the servers don't have replicas. no backups either. New: dovecot2 on a single huge server with 16TB (SATA) storage and a few SSDs we store fresh mails on the SSDs and run a doveadm purge to move mails older than a day to the SATA disks there is an identical server which has a max-15min-old rsync backup from the primary server higher-ups/management wanted to pack in as much storage as possible per server in order to minimise the cost of SSDs per server the rsync'ing is done because GlusterFS wasn't replicating well under that high small/random-IO. scaling out was expected to be done with provisioning another pair of such huge servers on facing disk-crunch issues like in the old architecture, manual moving of email accounts would be done. Concerns/doubts: I'm not convinced with the synchronously-replicated filesystem idea works well for heavy random/small-IO. GlusterFS isn't working for us yet, I'm not sure if there's another filesystem out there for this use case. The idea was to keep identical pairs and use DNS round-robin for email delivery and IMAP/POP3 access. And if one the servers went down for whatever reasons (planned/unplanned), we'd move the IP to the other server in the pair. In filesystems like Lustre, I get the advantage of a single namespace whereby I do not have to worry about manually migrating accounts around and updating MAILHOME paths and other metadata/data. Questions: What are the typical methods used to scale up/out with the traditional software (courier-imapd / dovecot)? Do traditional software that store on a locally mounted filesystem pose a roadblock to scale out with minimal "problems"? Does one have to re-write (parts of) these to work with an object-storage of some sort - such as OpenStack object storage?

  Read the article

• How can I determine Breaking point of my Web application using JMeter?

  - by Gopu Alakrishna

  How can I determine Breaking point of my Web application using JMeter? I have executed the JMeter Testplan with different concurrent users load. EX. 300 users(0% error), 400 users(7% error in a sample, 5% error in another sample), 500 users(more than 10% error in 4 out of 6 samples). At What value of % Error, I can say system reached the Breaking point.I used concurrent users 300, 400, 500 in a PHP website. Should I consider any other parameter to determine breaking point. How many maximum concurrent users my application can support?

  Read the article

• Apache stops serving requests when connections increase

  - by Gunjan

  The values for MaxClients, ServerLimit etc parameters are quite high (4000). Available RAM on the server is high too (~8G). Load average remains below 1 on a 24 core CPU. But when the number of visitors on the website increase apache just stops serving requests. The apache error log is blank and access log shows no more requests coming in. Restarting apache makes it work again until the number of requests increases again. Any ideas where to start looking? UPDATE Getting the below errors in apache error log on running it with LogLevel Debug [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 479 idle, and 1027 total children

  Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >