Monday, April 16, 2007

So few really need uptime

Here's my latest take on high availability:

1. Very, very few really need it.
2. The vast majority can live with being down for days.
3. Those who really, truly need it must be able to spend a lot of everything on it - continutally.
4. Those who buy HA-gear (software and hardware) without realising how much they have to spend on it will suffer downtime much worse than without the HA-gear.

Pity those folks who buy the gear and think it works like advertised, out of the box, like the vendor said, like the POC showed, and so on. They're the true victims.

Typically, it takes 8-9 months to truly test and stabilise a RAC system. As I've said somewhere else, some people elect to spend all of those nine months before going production whereas others split it so that some of the time is spent before and, indeed, some of it after going production.

But that's not all: Even when the system has been stabilised and runs fine, it will a couple of times a year or more often go down and create problems that you never saw before.

It's then time to call in external experts, but instead of just fixing the current cause of your IT crisis, I'd like to suggest that you instead consider the situation as one where you need to spend a good deal of resources in stabilising your system again - until the next IT crisis shows up.

Your system will never be truly stable when it's complex. The amount of effort and money you'll need to spend on humans being able to react to problems, running the system day-to-day, and - very important - keep them on their toes by having realistic (terribly expensive) test systems, courses, drills on realistic gear, networks of people who can help right now, and so forth... is huge.

The ironic thing is this: If you decide that you can live with downtime, and therefor with a much less complex system - your uptime will increase. Of course.

Mark Souza of Microsoft said it: Complexity is the enemy of availability.

44 Comments:

Anonymous Anonymous said...

I totally agree.

Dominic Delmolino
www.oraclemusings.com

11:53 AM  
Blogger Moans Nogood said...

And you should know!

For those of you who don't know Dom, he was one of the pillars (2nd in command, or so) of Oracle's System Performance Group under Cary Millsap. Dom was the first human being on the Planet to master Symmetric Replication (as it was called).

5:49 AM  
Anonymous Anonymous said...

But don't all added features increase complexity at some level? I once posted that if you are 100% averse to complexity you should be using dbopen(3),recno(3),hash(3) (and family ) on flat files, not Oracle. Or maybe C-ISAM?

Sure, the "features" that enable RAC functionality are a lot more visible than the underpinnings of, say, data guard but both are very complex. No?

10:55 AM  
Anonymous Anonymous said...

I totally agree too.

We used to have an (dual) OPS instance way back from 1996-1999: Oracle7.

In the end, it *definitely* gave us more downtime than it promised us uptime.
We're back to a single instance now: way more stable!

I remember one particular situation where both OPS instances would crash if a stored procedure was deployed on instance A, that would (temporarily) invalidate another stored pl/sql module that some session would still have a shared pool lock on at the other instance.

Oracle never got around to fixing that one for us.
Wonder if it's still there.
We had a couple of query scripts that needed to be run prior to delivering a stored procedure, just to check for a (known to us) situation that would crash the lot. And sometimes we had to just opt to bringing down one instance, do the deploy, and re-integrate the other instance again.

And I'm not talking about the unexplained freezes that would occur when the instances were running for a long time (that is, 2-3 weeks). Yep, proactive bounces in the weekend were the cure.
We (the dba-team) used to joke, that Larry's software was starting to look a lot like Bill's.

Toon

1:38 PM  
Blogger Moans Nogood said...

Yeah, everything is complex if you dive deep enough into it, I guess.

A fair question to ask these days could for instance be: Is a relational database really needed for this or that data and purpose? And another could be: Would it be OK to lose data?

We've been brought up in the RDBMS school to think the obvious and only answer is Yes to both questions, but we shouldn't let that cloud our judgement of the real needs of customers.

When something is not really needed, although it would be cool if it worked, you've got the cart before the horse.

A simple backup of your system, where restore and recovery is tested regularly, is what most people really need. They don't need DR-sites, DataGuard, and such.

For political reasons you might have to implement all sorts of things, and I still haven't found an effective way of preventing that from happening.

Toon's story about OPS reminds me of the early 90's where an airport was running OPS to make sure the passengers could find their gates.

The system would crash (cluster, OPS, the works) two-four times a year, which was pretty bad since nobody knew where to go then.

Every time I'd look and look at logs, dumps, traces, bug texts, TAR's (called PMS, and such back then) without finding the reason for the crash. The hardware vendor would do the same.

We were probably not good enough back then, either, but we tried for a couple of years.

Then I suggested they just run on one node, ie. stop using OPS, and they've been stable ever since.

No, OPS is not RAC, but similar situations exist aplenty to prove my point that it takes SO much effort and dedication to get REAL high-availability.

I should point out to you a few guys who have been very inspirational for me in this area:

James Morle with his "Disposable Computing Architecture" (if it breaks, throw it away and get the spare from the cuppard).

Jeff Needham, who introduced me to the notion that a DL-585 from HP in fact IS your HA - and also introduced me to the term "moving parts".

Mark Souza, head of Customer Advisory Team at MS, for the quote "Complexity is the enemy of availability" (or something like it).

6:42 PM  
Anonymous Anonymous said...

Excellent point, most 'standard' hardware these days is so redundant anyway - multiples of everything, hot swappable stuff, etc that the odds of the server going down are pretty small. The odds of some BOFH doing something bad to the data that does not get picked up for days or a drunk sysadmin/DBA (like me) doing an rm -r in the wrong directory are a lot higher too me thinks. I also worry about the storage system that RACs and stuff run on, there's usually only one of them to share between the servers unless there is a spare that is being mirrored to. What if a repair guy forgets his radioactive pace maker inside there for a few days? The KISS principle seems to applicable here too - Keep It Simple Stupid

10:05 PM  
Anonymous Gavin said...

You are a owner of such a nice blog. And also it has got more than 60 indexed posts. So I am little surprised with the negative title used for it. Don't you do blogs regularly!

10:13 PM  
Blogger Moans Nogood said...

It's referring to a website I created several years ago called Vi Bruger Ikke (vibrugerikke.dk) which was later translated into wedonotuse.com.

So it's just a joke calling it we do not use blogs.

Mogens

10:23 PM  
Blogger Andrew said...

Hi Mogens

I did like your take on this. Conversely - those customers who do have the discipline to put a cost on their downtime also often have the discipline and savvy to really make RAC (and other resilience technologies) really work - and they really are great to work with...

Kind regards
Andrew (back in touch after some years...)
http://blogs.oracle.com/asparks

7:24 AM  
Anonymous Guide for World of Warcraft Gold said...

great post!

10:48 AM  
Blogger Moans Nogood said...

Hi Andrew,

Ah, good to have you back. What have you been doing?

Mogens

3:59 PM  
Anonymous polo jacket said...

Articles are meaningful, and your blog is nice!
polo jacket
polo shirt
ralp lauren jacket
cheap polo shirts
spyder jackets
discount polo shirts
ralph lauren shirt
columbia jacket
north face jacket
cheap ralph lauren shirts
women's columbia jackets
polo t shirts
polo men's shirt
ralph lauren men's shirt
polo mens shirt
cheap polo jackets
ralph lauren mens shirt
wholesale polo jacket
ralp lauren polo shirts
short sleeve polo shirt
men polo shirt
lacoste polo shirts
wholesale polo shirts
men's polo shirts
cheap polo ralph lauren
cheap polo t-shirts
cheap polo clothes
custom polo shirts
discount north face jackets
ladies spyder jacket
tennis rackets
tennis racket
tennis racquet
tennis racquets
wilson tennis rackets
prince tennis rackets
head tennis rackets
babolat tennis rackets
best tennis racket
cheap tennis rackets
wilson tennis racquets
head tennis racquets
babolat tennis racquets
cheap tennis racket

2:56 AM  
Anonymous polo shirt said...

Articles are meaningful, and your blog is nice!
polo jacket
polo shirt
ralp lauren jacket
cheap polo shirts
spyder jackets
discount polo shirts
ralph lauren shirt
columbia jacket
north face jacket
cheap ralph lauren shirts
women's columbia jackets
polo t shirts
polo men's shirt
ralph lauren men's shirt
polo mens shirt
cheap polo jackets
ralph lauren mens shirt
wholesale polo jacket
ralp lauren polo shirts
short sleeve polo shirt
men polo shirt
lacoste polo shirts
wholesale polo shirts
men's polo shirts
cheap polo ralph lauren
cheap polo t-shirts
cheap polo clothes
custom polo shirts
discount north face jackets
ladies spyder jacket
tennis rackets
tennis racket
tennis racquet
tennis racquets
wilson tennis rackets
prince tennis rackets
head tennis rackets
babolat tennis rackets
best tennis racket
cheap tennis rackets
wilson tennis racquets
head tennis racquets
babolat tennis racquets
cheap tennis racket

5:03 AM  
Anonymous John said...

Buzz words and hype cause more problems than they solve (including unnecessary complexity)...

BUT there are a number of devices that you can buy that can provide redundancy with a minimum of complexity... (e.g. a dual wan router from Draytek, RAID1 NAS, etc.)

So map out WHERE you are most likely to have a failure (and where you're most likely to suffer if you have a failure).

John's Linux Tutorials (Pfeiffer)


John's SEO and Web Tutorials (Pfeiffer)

3:24 AM  
Anonymous Anonymous said...

Your blog is very good, it's the best blog I've ever seen, thank you!
By the way, do you like polo shirts, which are very chic, especially the polo t shirts, I love them very much. I also like playing tennis rackets, it can keep healthy, what do you like to do? I'd like to introduce myself as the outlet of polo t shirts women, polo t shirts on sale, and polo t shirts for women. These products are our masterpieces, such as polo shirts on sale, polo shirts men, men's polo shirt, men polo shirt, mens polo shirts, mens polo shirt, besides we also sell cheap polo shirts, discount polo shirts, men's polo shirts, women's polo shirts. We are also the outlet of, cheap tennis racket, discount tennis racket, we recommend prince tennis racquet, head tennis rackets, wilson tennis racket, babolat tennis racquet. And it is our great pleasure that you come to our store online!

6:48 PM  
Blogger ai said...

polo boots
It's all about fierce glamour with high octane gloss and lashings of sparkle as fabrics go metallic with shimmering luxe finishes. Forpolo shoes
, gloriously excessive embellishment is absolutely key, championed at cheap herve leger outlet
and Elie Saab.

7:23 PM  
Blogger Unknown said...

Ed hardy streak of clothing is expanded into its wholesale ED Hardy chain so that a large number of fans and users can enjoy the cheap ED Hardy Clothes range easily with the help of numerous secured websites, actually, our discount ED Hardy Outlet. As we all know, in fact Wholesale Ed Hardy,is based on the creations of the world renowned tattoo artist Don Ed Hardy. Why Ed hardy wholesale? Well, this question is bound to strike the minds of all individuals. Many people may say cheap Prada shoes is a joke, but we can give you discount Prada Sunglasses , because we have authentic Pradas bags Outlet. Almost everyone will agree that newest Pradas Purses are some of the most beautiful designer handbags ( Pradas handbags 2010) marketed today. Now we have one new product: Prada totes. The reason is simple: fashion prohibited by ugg boots, in other words, we can say it as Cheap ugg boots or Discount ugg boots. We have two kinds of fashionable boots: classic ugg boots and ugg classic tall boots. Ankh Royalty--the Cultural Revolution. Straightens out the collar, the epaulette epaulet, the Ankh Royalty Clothing two-row buckle. Would you like to wear Ankh Royalty Clothes?Now welcome to our AnkhRoyalty Outlet. And these are different products that bear the most famous names in the world of fashion, like Ankh Royalty T-Shirt, by the way-Prada, Spyder, Moncler(Moncler jackets,or you can say Moncler coats, Moncler T-shirt, Moncler vest,and you can buy them from our discount Moncler outlet), GHD, ED Hardy, designer Sunglasses, Ankh Royalty, Twisted Heart.

11:14 PM  
Blogger combattery84 said...

SONY VGP-BPS5 battery
FUJITSU Fpcbp80 Battery
FUJITSU Fpcbp83 Battery
FUJITSU Fpcbp95 Battery
FUJITSU Fpcbp98 Battery
FUJITSU Fpcbp121 Battery
FUJITSU Fpcbp151 Battery
FUJITSU lifebook t4010 Battery
FUJITSU lifebook t4020d Battery
TOSHIBA PA3307U-1BRS Battery
TOSHIBA PA3383U-1BRS Battery
TOSHIBA PA3384U-1BRS Battery ACER TRAVELMATE 4600 Battery
HP PAVILION DV4000 Battery
HP PAVILION DV9000 Battery
TOSHIBA PA3465U-1BRS Battery
Compaq Presario R3000 Battery HP F4809A Battery
Dell INSPIRON 6000 Battery
IBM THINKPAD T42 Battery
Dell Latitude D610 Battery
SONY VGP-BPS2 Battery
APPLE A1078 Battery
APPLE M8403 Battery <a

8:33 PM  
Blogger combattery84 said...

Acer ASPIRE 1300 battery
Acer ASPIRE 1310 battery
Acer Aspire 1410 battery
Acer ASPIRE 1680 battery
ACER BTP-63D1 battery ACER BTP-43D1 battery
Acer lc.btp05.001 battery
Compaq Business Notebook NX9000 series battery
FUJITSU Lifebook C2220 battery
FUJITSU Fpcbp63 Battery
FUJITSU Fpcbp68 Battery FUJITSU Fpcbp77 Battery
FUJITSU Fpcbp78 Battery
FUJITSU Fpcbp79 Battery
Armada E700 Series battery
SONY PCGA-BP1N battery
SONY PCGA-BP2E battery
SONY PCGA-BP2NX battery
SONY PCGA-BP2S battery
SONY PCGA-BP2SA battery
SONY PCGA-BP2T battery
SONY PCGA-BP2V battery
SONY PCGA-BP4V battery
SONY PCGA-BP71 battery
SONY PCGA-BP71A battery
SONY VGP-BPL1 battery
SONY VGP-BPL2 battery
HP F4098A battery

8:34 PM  
Blogger combattery84 said...

APPLE A1078 battery
APPLE A1079 battery
APPLE A1175 battery
APPLE A1189 battery
Business Notebook NX9110 Series battery
GATEWAY NX7000 Series laptop battery
IBM 02K6821 battery
IBM 02K7054 battery
IBM 08K8195 battery
IBM 08K8218 battery IBM Thinkpad 390 Series battery
HP omnibook xe3 battery
IBM Thinkpad 390X battery
IBM ThinkPad G40 Series battery
ThinkPad G41 Series battery
Toshiba PA2487UR battery
Toshiba Satellite A105 battery
Toshiba A70 battery
PA3062U-1BAT battery
Toshiba Satellite P30 battery
Toshiba PA3084U-1BRS battery
Toshiba PA3098U battery
PA3107U-1BAS battery
PA3107U-1BRS battery
PA3166U-1BRS battery
PA3176U-1BAS batteryPanasonic CGR-S006E battery Panasonic CGA-S006 battery Panasonic Lumix DMC-FZ8 battery OLYMPUS LI-40B battery <a

8:35 PM  
Anonymous Anonymous said...

Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery
dell Latitude C640 battery
Dell XPS M140 battery
Dell Inspiron E1405 battery
dell 700m battery
dell C1295 battery
Dell U4873 Battery
Dell Latitude C600 battery
Armada E700 Series battery
Compaq 116314-001 battery
Compaq 319411-001 battery
Compaq nc4200 battery
Compaq Presario R3000 Battery
Compaq Presario 2100 battery
Compaq Presario r3000 Battery
Compaq Business Notebook NX9000 series battery
HP 395789-001 battery
HP 446506-001 Battery
HP dv9700 battery
HP F4809A Battery
HP nc8000 battery
HP nc8230 battery

2:18 AM  
Anonymous Anonymous said...

HP pavilion zd8000 battery
HP f2024b battery
HP f4812a battery
HP Pavilion ZV5000 battery
HP Pavilion DV1000 battery
HP Pavilion ZD7000 Battery
HP Pavilion DV2000 battery
HP Pavilion DV4000 Battery
HP Pavilion dv6000 Battery
HP Pavilion DV9000 Battery
HP F4098A battery
HP pavilion zx6000 battery
HP omnibook xe4400 battery
HP omnibook xe4500 battery
HP omnibook xe3 battery
Notebook NX9110 battery
IBM 02K6821 battery
IBM 02K7054 battery
IBM 08K8195 battery
IBM 08K8218 battery
IBM 92P1089 battery
IBM Thinkpad 390 Series battery
IBM Thinkpad 390X battery
IBM ThinkPad Z61m Battery
IBM 02K7018 Battery
IBM thinkpad t41p battery
IBM THINKPAD T42 Battery
IBM ThinkPad R60 Battery
IBM ThinkPad T60 Battery
IBM ThinkPad T41 Battery

2:20 AM  
Anonymous Anonymous said...

Toshiba Satellite A105 battery
Toshiba A70 battery
PA3062U-1BAT battery
Toshiba Satellite P30 battery
Toshiba PA3084U-1BRS battery
Toshiba PA3098U battery
PA3107U-1BAS battery
PA3107U-1BRS battery
PA3166U-1BRS battery
PA3176U-1BAS battery
TOSHIBA PABAS076 Battery
Toshiba pa3399u-1brs battery
TOSHIBA PA3399U-2BAS Battery
TOSHIBA PA3421U-1BRS Battery
TOSHIBA PA3456U-1BRS Battery
TOSHIBA Pa3356u-1brs battery
Satellite a10 battery
Pa3331u-1brs battery
Satellite m30 series battery
Satellite pro m30 battery
TOSHIBA PA3399U-1BRS Battery
Portege m300 battery
TOSHIBA PA3285U-1BRS Battery
Canon BP-2L5 Battery
Canon BP-508 Battery
JVC BN-VF707U Battery
JVC BN-VF707 Battery

2:23 AM  
Anonymous Anonymous said...

dhdt

3:36 AM  
Blogger winnie said...

moncler jacke
outlet moncler
moncler shoes
Moncler

Bandage dress
Herve Leger dresses
Herve Leger

dsquared jeans

12:44 AM  
Blogger Wholesale NFL Jerseys said...

Anaheim Ducks
Atlanta Thrashers
Boston Bruins
Buffalo Sabres
Calgary Flames
Carolina Hurricanes
Chicago Blackhawks
Colorado Avalanche
Columbus Blue Jackets
Dallas Stars
Detroit Red Wings
Edmonton Oilers
Los Angeles Kings
Minnesota Wild
Montreal Canadiens
Nashville Predators
New Jersey Devils
New York Islanders
New York Rangers
Olympic Canada
Olympic USA

11:26 PM  
Blogger Wholesale NFL Jerseys said...

Good post! Thanks for your information! ,ASICS Running shoes,Onitsuka Shoes-ASICS Tiger shoes
Shop for Onitsuka Tiger,ASICS Shoes,Asics Tiger Shoes from Onitsukatiger.us.Full range of Onitsuka Tiger Shoes for Men and Women,Free Shipping!
www.Onitsukatiger.us

new arrival
asics revolve le
asics top seven
onitsuka tiger womens
asics whizzer lo womens
onitsuka tiger mini cooper kids
onitsuka tiger mexico mid runner
onitsuka tiger mexico 66
onitsuka tiger mexico 66 lauta
onitsuka tiger mexico 66 deluxe
onitsuka tiger mexico 66 limited
onitsuka tiger mini cooper
onitsuka tiger mexico 66 baja
onitsuka tiger olympos
onitsuka tiger rotation 77
onitsuka tiger ultimate 81
onitsuka tiger tokidoki
onitsuka tiger california 78
onitsuka tiger x tokidoki fabre
asics running gel-stratus 2.1
asics running gel-kinsei 2
asics running gel lyte iii
asics suroingu japan ar

10:33 PM  
Anonymous Anonymous said...

For thosepink chi of you who have not been on the show Hair, people watching is fantastic. It's a world where the rules of hair statement. We found, Rockabilly Pony, Cheap GHDteased bouffant and crazy on the color and

was public. cheap ugg boots clearance shoesFor a publisher more accustomed to moderate hair of Toronto, was a visual perception.

2:26 AM  
Anonymous Anonymous said...

I ardently recognize that the know-how supplied is related to most people . Appreciate it .
Riverside CA Locksmith
Locksmith Wellington
Cupertino locksmith
Milpitas locksmith
Milpitas locksmith
Milpitas locksmith
Locksmith Milpitas CA
Locksmith Milpitas CA
Locksmith Glenview
Locksmith Glenview IL
Locksmith Glenview
Locksmith Glenview
Locksmith Glenview IL
Glenview Locksmith
Locksmith Glenview
Locksmith Glenview IL
Locksmith Glenview
Locksmith Glenview IL
Glenview Locksmith
Glenview Locksmith
Locksmith Glenview
Locksmith Glenview
plano tx locksmith
mesquite locksmith
mesquite locksmith
mesquite locksmith
locksmith miami beach fl
plano tx locksmith
plano tx locksmith
plano tx locksmith
plano tx locksmith

2:59 AM  
Anonymous Penis Enlargement Pills said...

Really you have done great job,There are may person searching about that now they will find enough resources by your post.I like this blog..

6:24 AM  
Anonymous VigRX Plus said...

I just added your web page to my favorites. I really enjoy reading your posts. Thank you!

6:27 AM  
Anonymous Anonymous said...

black boots

Chocolate boots

sand boots

Chestnut boots

gray boots

pink boots

grey boots

Ugg Boots For Sale

Sand Ugg Boots

1:20 AM  
Blogger Frank said...

Hey,
Loving your blog, awesome tips on blog you have here. I
would just like to ask you some questions privately, mindWinRaR Free Download,IDM Free Download,function venues geelong,la Fitness,Mobile Prices in Pakistan

3:04 PM  
Anonymous Anonymous said...

Online Cricket Score
Fifa News

6:17 PM  
Blogger Alberta Plumuj said...

UGG Adirondack II Boots
UGG Adirondack Tall Boots
UGG And Jimmy Choo Boots
UGG Argyle Knit Boots
UGG Ashur Tall Boots
UGG Bailey Button Boots
UGG Brookfield Tall Boots
UGG Classic Cardy Boots
UGG Classic Mini Boots
UGG Classic Short Boots
UGG Classic Tall Boots
UGG Classic Tall Metallic Boots
UGG Classic Tall Stripe Cable Knit Boots
UGG Corinth Boots
UGG Elsey Boots
UGG Felicity Boots
UGG Gissella Boots
UGG Highkoo Amber Boots
UGG Kensington Short Boots
UGG Kids Boots
UGG Langley Boots
UGG Liberty Boots
UGG Nightfall Boots
UGG Oliviya Boots
UGG Payton Boots
UGG Roseberry Tall Boots
UGG Sandra Boots
UGG Seline Boots
UGG Slippers
UGG Suburb Crochet Tall Boots
UGG Sundance II Boots
UGG Ultra Short Boots
UGG Ultra Tall Boots

11:06 PM  
Blogger Mohtasham said...

Man I must say your mind works regarding all the this. I learned alot from your article keep teaching us more. I really appreciate you and want to talk you please tell me how can contact you? how to cook biryani l latest sms l la fitness

3:28 AM  
Blogger Ghd Australia said...

Uggs Outlet
Uggs Outlet Online
Uggs Outlet Stores

6:55 PM  
Anonymous Anonymous said...

Watchmaking continues to be powerful simply because it's a area which has revolutionized by itself through inside. beats headphones It is among enchanting craftwork, method, appearance, style, developments as well as economics. Watchmakers possess been successful for making the actual view a really fascinating item. This has turned into a interpersonal indication. A wrist watch nevertheless informs time. It's main perform hasn't vanished. However it has additionally turn out to be a good item. monster beats These days, you could state: "Show me personally your own view, as well as I will let you know that you're. inch Self-conscious individuals can't disregard the view like a design declaration.

1:31 AM  
Blogger oakleyses said...

oakley sunglasses, michael kors outlet, louboutin outlet, prada handbags, air max, burberry outlet, louis vuitton outlet, ray ban sunglasses, tory burch outlet, cheap uggs, michael kors outlet, michael kors outlet, chanel handbags, burberry outlet, oakley sunglasses, kate spade outlet, uggs outlet, oakley sunglasses cheap, prada outlet, longchamp handbags, gucci outlet, air max, louis vuitton, ray ban sunglasses, nike shoes, jordan shoes, nike free, polo ralph lauren, michael kors outlet, polo ralph lauren outlet, ray ban sunglasses, oakley sunglasses, michael kors outlet, christian louboutin, tiffany and co, louboutin, tiffany and co, rolex watches, longchamp outlet, louis vuitton outlet, longchamp handbags, ugg boots clearance, michael kors outlet online sale, uggs, ugg outlet, louis vuitton outlet stores, louboutin shoes, replica watches, louis vuitton handbags

4:49 PM  
Blogger oakleyses said...

converse, pandora charms, toms shoes, barbour, moncler outlet, sac louis vuitton, swarovski, links of london, louis vuitton, barbour, moncler, canada goose, moncler, canada goose jackets, pandora charms, converse shoes, rolex watches, vans, supra shoes, moncler, ugg boots, abercrombie, ugg, marc jacobs, ugg pas cher, ugg, wedding dresses, pandora jewelry, sac lancel, air max, louis vuitton, sac louis vuitton, coach outlet store online, juicy couture, hollister, ugg, canada goose, canada goose, moncler, juicy couture outlet, thomas sabo, moncler, ray ban, louis vuitton uk, montre homme, moncler, gucci, canada goose outlet, pandora jewelry, canada goose, canada goose uk

4:56 PM  
Blogger Unknown said...

adidas nmd
cheap jordan shoes
ugg outlet
oakley sunglasses
louis vuitton handbags
polo ralph lauren
ugg boots
polo ralph lauren
christian louboutin uk
ugg boots
201612.9wengdongdong

7:57 PM  
Blogger Unknown said...

https://decor-ksa.com/
http://mchaabaty.com/dawadmi/
http://mchaabaty.com/templates/
http://mchaabaty.com/riyadh/
http://mchaabaty.com/stone/

10:47 AM  
Blogger Unknown said...

qzz0606
coach outlet canada
saucony shoes
reebok outlet store
air max 90
nike factory outlet
y3 shoes
michael kors outlet
canada goose outlet
pandora charms
canada goose outlet

2:35 AM  
Blogger yanmaneee said...

curry 4
air max 97
retro jordans
moncler jackets
yeezy boost 350
fila shoes
golden goose sneakers
nike 97
michael kors handbags
chrome hearts

9:39 PM  

Post a Comment

<< Home